LustreError: 9549:0:(lraft_lib.c:831:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9549:0:(lraft_lib.c:831:lraft_init()) LBUG
CPU: 0 PID: 9549 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9f/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-150vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-150vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-150vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-150vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-150vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-150vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-150vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-150vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.69@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 391d57f3-024e-4129-b5e4-02420a1b9d6c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 391d57f3-024e-4129-b5e4-02420a1b9d6c.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.69@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem d0c0e4e8-b827-4af0-9f3c-226ebae0ac58 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem d0c0e4e8-b827-4af0-9f3c-226ebae0ac58.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-150vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-150vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 391d57f3-024e-4129-b5e4-02420a1b9d6c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 391d57f3-024e-4129-b5e4-02420a1b9d6c.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_3_g7e1cffa
LNet: Added LNI 10.240.27.70@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 391d57f3-024e-4129-b5e4-02420a1b9d6c r/w with ordered data mode. Quota mode: journalled.
Lustre: 9355:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.27.69@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9551:0:(lraft_lib.c:831:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9551:0:(lraft_lib.c:831:lraft_init()) LBUG
CPU: 0 PID: 9551 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9f/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.29@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 29952366-755f-4468-b47f-558d060b5812 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 29952366-755f-4468-b47f-558d060b5812.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.29@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 486bdfe6-ea22-41bb-9d35-1687da7447f5 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 486bdfe6-ea22-41bb-9d35-1687da7447f5.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 29952366-755f-4468-b47f-558d060b5812 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 29952366-755f-4468-b47f-558d060b5812.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_3_g7e1cffa
LNet: Added LNI 10.240.27.30@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 29952366-755f-4468-b47f-558d060b5812 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9357:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.27.29@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9529:0:(lraft_lib.c:831:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9529:0:(lraft_lib.c:831:lraft_init()) LBUG
CPU: 1 PID: 9529 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9f/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-131vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-131vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-131vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-131vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-131vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-131vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-131vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-131vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.45.71@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem b5138bf7-e712-4bda-8d76-28747987942b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem b5138bf7-e712-4bda-8d76-28747987942b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.45.71@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 5dd5e56c-cee0-422d-892c-d27d347866cf r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 5dd5e56c-cee0-422d-892c-d27d347866cf.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-131vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-131vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem b5138bf7-e712-4bda-8d76-28747987942b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem b5138bf7-e712-4bda-8d76-28747987942b.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_3_g7e1cffa
LNet: Added LNI 10.240.45.72@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem b5138bf7-e712-4bda-8d76-28747987942b r/w with ordered data mode. Quota mode: journalled.
Lustre: 9335:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.45.71@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9552:0:(lraft_lib.c:831:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9552:0:(lraft_lib.c:831:lraft_init()) LBUG
CPU: 0 PID: 9552 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9f/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-140vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.201@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem b30ce553-e557-470c-a77c-3c1284fdede1 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem b30ce553-e557-470c-a77c-3c1284fdede1.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.201@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 0dea07d0-4b1b-48b7-ac3e-953579c536eb r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 0dea07d0-4b1b-48b7-ac3e-953579c536eb.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem b30ce553-e557-470c-a77c-3c1284fdede1 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem b30ce553-e557-470c-a77c-3c1284fdede1.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_3_g7e1cffa
LNet: Added LNI 10.240.25.202@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem b30ce553-e557-470c-a77c-3c1284fdede1 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9358:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.25.201@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10193:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10193:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 10193 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.141@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 119a3322-f576-470c-83d8-6decc9b41725 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 119a3322-f576-470c-83d8-6decc9b41725.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.141@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem cfcc1771-d277-4e3b-be46-c002c859ba96 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem cfcc1771-d277-4e3b-be46-c002c859ba96.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_25791.112)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.42.142@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-78vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 119a3322-f576-470c-83d8-6decc9b41725 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 119a3322-f576-470c-83d8-6decc9b41725.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 119a3322-f576-470c-83d8-6decc9b41725 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10016:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.42.141@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 11321:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 11321:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 11321 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm3.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm2.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm3.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: trevis-76vm2.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
LNet: Added LNI fd33:3981:3213:f020:0:5254:3e:2958@tcp [8/256/0/180]
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f020:0:5254:17:7c7f@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-0): mounted filesystem 10b10681-4537-435e-84b7-79652959a2a7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 10b10681-4537-435e-84b7-79652959a2a7.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f020:0:5254:17:7c7f@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-1): mounted filesystem c8605323-defd-499a-8007-745b0258d942 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem c8605323-defd-499a-8007-745b0258d942.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm2.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm3.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm2.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-76vm3.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 10b10681-4537-435e-84b7-79652959a2a7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 10b10681-4537-435e-84b7-79652959a2a7.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 10b10681-4537-435e-84b7-79652959a2a7 r/w with ordered data mode. Quota mode: journalled.
Lustre: 11287:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGCfd33:3981:3213:f020:0:5254:17:7c7f@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9547:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9547:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 9547 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-82vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-82vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-82vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-82vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-82vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-82vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-82vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-82vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.224@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 47b13c11-ac96-42d4-98ba-ed1fa2d3eca3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 47b13c11-ac96-42d4-98ba-ed1fa2d3eca3.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.224@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 71e8d1a6-d2af-4e26-bd8c-c02718b43e25 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 71e8d1a6-d2af-4e26-bd8c-c02718b43e25.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-82vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-82vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 47b13c11-ac96-42d4-98ba-ed1fa2d3eca3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 47b13c11-ac96-42d4-98ba-ed1fa2d3eca3.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.26.225@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 47b13c11-ac96-42d4-98ba-ed1fa2d3eca3 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9353:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.26.224@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9558:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9558:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 9558 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-72vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-72vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-72vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-72vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-72vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-72vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-72vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-72vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.24@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem dfd7ca9f-116f-4a69-99ad-27769fb6be34 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem dfd7ca9f-116f-4a69-99ad-27769fb6be34.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.24@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 01ffa9ba-be09-41d5-a797-11a77844472e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 01ffa9ba-be09-41d5-a797-11a77844472e.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-72vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-72vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem dfd7ca9f-116f-4a69-99ad-27769fb6be34 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem dfd7ca9f-116f-4a69-99ad-27769fb6be34.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.26.25@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem dfd7ca9f-116f-4a69-99ad-27769fb6be34 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9364:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.26.24@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9555:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9555:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 9555 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-75vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-75vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-75vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.84@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 66000157-1831-4abc-8572-5d8ca9b9cb9b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 66000157-1831-4abc-8572-5d8ca9b9cb9b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.84@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem ce705567-910e-44f8-96d1-a533895d1993 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem ce705567-910e-44f8-96d1-a533895d1993.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 66000157-1831-4abc-8572-5d8ca9b9cb9b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 66000157-1831-4abc-8572-5d8ca9b9cb9b.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.26.85@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 66000157-1831-4abc-8572-5d8ca9b9cb9b r/w with ordered data mode. Quota mode: journalled.
Lustre: 9361:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.26.84@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9559:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9559:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 9559 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-69vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-69vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-69vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-69vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-69vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-69vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-69vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-69vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.222@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 504b2db9-ce80-487c-809d-b9e34c4b0e6f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 504b2db9-ce80-487c-809d-b9e34c4b0e6f.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.222@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem e8ea5c34-12ae-46b8-ba27-678e3768f7a6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem e8ea5c34-12ae-46b8-ba27-678e3768f7a6.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-69vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-69vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 504b2db9-ce80-487c-809d-b9e34c4b0e6f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 504b2db9-ce80-487c-809d-b9e34c4b0e6f.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.25.223@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 504b2db9-ce80-487c-809d-b9e34c4b0e6f r/w with ordered data mode. Quota mode: journalled.
Lustre: 9366:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.25.222@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9570:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9570:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 9570 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-86vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-86vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-86vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-86vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-86vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-86vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-86vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-86vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.50@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem f058536d-59a2-4293-8279-c6bd951932a5 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem f058536d-59a2-4293-8279-c6bd951932a5.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.50@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 6652b650-56c9-42e3-a198-085449e6244b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 6652b650-56c9-42e3-a198-085449e6244b.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-86vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-86vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem f058536d-59a2-4293-8279-c6bd951932a5 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem f058536d-59a2-4293-8279-c6bd951932a5.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.27.51@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem f058536d-59a2-4293-8279-c6bd951932a5 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9376:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.27.50@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: Not available for connect from 10.240.27.50@tcp (not set up)
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9542:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9542:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 9542 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe98/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-94vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-94vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-94vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-94vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-94vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-94vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-94vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-94vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.205@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 3e649234-b4cc-4850-b1c0-e519182d07f9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 3e649234-b4cc-4850-b1c0-e519182d07f9.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.205@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 52598f54-2c57-4e82-a6fa-9bdb32df2bce r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 52598f54-2c57-4e82-a6fa-9bdb32df2bce.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-94vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-94vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 3e649234-b4cc-4850-b1c0-e519182d07f9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 3e649234-b4cc-4850-b1c0-e519182d07f9.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn36_2_gfb16386
LNet: Added LNI 10.240.43.206@tcp [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 3e649234-b4cc-4850-b1c0-e519182d07f9 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9348:0:(mgc_request_server.c:714:mgc_llog_local_copy()) MGC10.240.43.205@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10214:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10214:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 10214 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-76vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.101@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 46b64512-664c-4735-a4b6-f7ae1d1f7d78 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 46b64512-664c-4735-a4b6-f7ae1d1f7d78.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.101@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 31340a2b-a9aa-4c58-9ae2-dade22cf0895 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 31340a2b-a9aa-4c58-9ae2-dade22cf0895.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_25654.112)
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.42.102@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-76vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-76vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 46b64512-664c-4735-a4b6-f7ae1d1f7d78 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 46b64512-664c-4735-a4b6-f7ae1d1f7d78.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 46b64512-664c-4735-a4b6-f7ae1d1f7d78 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10038:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.42.101@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 10038:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10258:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10258:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 10258 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-28vm10.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-28vm10.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-28vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-28vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-28vm10.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-28vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-28vm10.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-28vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.168@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 28062234-4069-472a-a9e3-439f80cf389f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 28062234-4069-472a-a9e3-439f80cf389f.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.168@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 12ba0460-df10-40d6-9acb-93285452c6fe r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 12ba0460-df10-40d6-9acb-93285452c6fe.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_25654.111)
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 10 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_25654.111)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.38.169@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-28vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-28vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 28062234-4069-472a-a9e3-439f80cf389f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 28062234-4069-472a-a9e3-439f80cf389f.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 28062234-4069-472a-a9e3-439f80cf389f r/w with ordered data mode. Quota mode: journalled.
Lustre: 10082:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.38.168@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 10082:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9568:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9568:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9568 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.106@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 26fc51eb-2847-4249-8e9c-29598ea670df r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 26fc51eb-2847-4249-8e9c-29598ea670df.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.106@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 4e1cfee0-fb1f-4a61-90d0-5a914fe81ba6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 4e1cfee0-fb1f-4a61-90d0-5a914fe81ba6.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 26fc51eb-2847-4249-8e9c-29598ea670df r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 26fc51eb-2847-4249-8e9c-29598ea670df.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.38.107@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 26fc51eb-2847-4249-8e9c-29598ea670df r/w with ordered data mode. Quota mode: journalled.
Lustre: 9374:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.38.106@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9374:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9578:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9578:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9578 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-130vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-130vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-130vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-130vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-130vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-130vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-116vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.30.157@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 71a245f5-f6bc-4586-b925-5bf53fabfafd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 71a245f5-f6bc-4586-b925-5bf53fabfafd.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.30.157@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem c71d78d0-6c82-455c-913e-3f12df140363 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem c71d78d0-6c82-455c-913e-3f12df140363.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-130vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-130vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 71a245f5-f6bc-4586-b925-5bf53fabfafd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 71a245f5-f6bc-4586-b925-5bf53fabfafd.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.30.158@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 71a245f5-f6bc-4586-b925-5bf53fabfafd r/w with ordered data mode. Quota mode: journalled.
Lustre: 9384:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.30.157@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9384:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9618:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9618:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9618 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-99vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-99vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-99vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-116vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-99vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-99vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-99vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.46@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem a970590a-6e55-422b-8efb-034806ff42bc r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem a970590a-6e55-422b-8efb-034806ff42bc.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.46@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 9600bde3-5228-40d4-b2f7-16ce383f22dd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 9600bde3-5228-40d4-b2f7-16ce383f22dd.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-99vm3.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-99vm3.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem a970590a-6e55-422b-8efb-034806ff42bc r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem a970590a-6e55-422b-8efb-034806ff42bc.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.28.47@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem a970590a-6e55-422b-8efb-034806ff42bc r/w with ordered data mode. Quota mode: journalled.
Lustre: 9424:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.28.46@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9424:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10618:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10618:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 10618 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-124vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-124vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-124vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.30.39@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1317710 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 633b24d9-76b9-48bc-a782-7a8f1b124c86 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 633b24d9-76b9-48bc-a782-7a8f1b124c86.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.30.39@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1317710 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 9ef205ac-3eba-4ce5-8a40-effb16d3e414 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 9ef205ac-3eba-4ce5-8a40-effb16d3e414.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.30.39@tcp --fsname=lustre --mdt --index=5 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1317710 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt6
LDISKFS-fs (dm-2): mounted filesystem f948c316-f7d3-4779-b555-6b972b338559 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-2): unmounting filesystem f948c316-f7d3-4779-b555-6b972b338559.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds6' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-124vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 2637824 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-4): mounted filesystem 633b24d9-76b9-48bc-a782-7a8f1b124c86 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-4): unmounting filesystem 633b24d9-76b9-48bc-a782-7a8f1b124c86.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.30.40@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-4): mounted filesystem 633b24d9-76b9-48bc-a782-7a8f1b124c86 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10424:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.30.39@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 10424:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 11375:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 11375:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 11375 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-103vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-56vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm6.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-56vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm9.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm6.onyx.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: onyx-56vm9.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
LNet: Added LNI fd33:3981:3213:f010:0:5254:13:6ff9@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f010:0:5254:51:5b50@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-0): mounted filesystem a1914829-89a5-4a90-84e1-d2a1390fcd80 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem a1914829-89a5-4a90-84e1-d2a1390fcd80.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f010:0:5254:51:5b50@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-1): mounted filesystem 653f9443-9b96-4b5b-b0c0-8ccf29c44489 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 653f9443-9b96-4b5b-b0c0-8ccf29c44489.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm6.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm9.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm7.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-103vm6.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-56vm9.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-56vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-103vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-103vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem a1914829-89a5-4a90-84e1-d2a1390fcd80 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem a1914829-89a5-4a90-84e1-d2a1390fcd80.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem a1914829-89a5-4a90-84e1-d2a1390fcd80 r/w with ordered data mode. Quota mode: journalled.
Lustre: 11341:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGCfd33:3981:3213:f010:0:5254:51:5b50@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 11341:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9593:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9593:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9593 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-116vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.209@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 58ae1742-04cc-44de-a2e7-457f0607dbc4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 58ae1742-04cc-44de-a2e7-457f0607dbc4.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.209@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f7c38b2c-fbdc-440f-b44f-bd3f300ac8cb r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f7c38b2c-fbdc-440f-b44f-bd3f300ac8cb.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-107vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 58ae1742-04cc-44de-a2e7-457f0607dbc4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 58ae1742-04cc-44de-a2e7-457f0607dbc4.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.28.210@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 58ae1742-04cc-44de-a2e7-457f0607dbc4 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9399:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.28.209@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9399:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9541:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9541:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9541 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-32vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-32vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-32vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-32vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-32vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-32vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-32vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-32vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.245@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 679bdac5-ed30-4f24-9b5e-51f6a354e141 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 679bdac5-ed30-4f24-9b5e-51f6a354e141.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.245@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 061504cd-0d6f-4da8-b268-ad76bac99cb4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 061504cd-0d6f-4da8-b268-ad76bac99cb4.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-32vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-32vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 679bdac5-ed30-4f24-9b5e-51f6a354e141 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 679bdac5-ed30-4f24-9b5e-51f6a354e141.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.22.246@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 679bdac5-ed30-4f24-9b5e-51f6a354e141 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9366:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.22.245@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9366:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9597:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9597:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9597 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.207@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem f1e678b0-b684-4192-b6a2-3f9304ef0f1b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem f1e678b0-b684-4192-b6a2-3f9304ef0f1b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.207@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 130db1f0-43f8-43a2-bbe5-cb1b2f522d3d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 130db1f0-43f8-43a2-bbe5-cb1b2f522d3d.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-107vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem f1e678b0-b684-4192-b6a2-3f9304ef0f1b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem f1e678b0-b684-4192-b6a2-3f9304ef0f1b.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.28.208@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem f1e678b0-b684-4192-b6a2-3f9304ef0f1b r/w with ordered data mode. Quota mode: journalled.
Lustre: 9403:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.28.207@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9403:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9566:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9566:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9566 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-59vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-59vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-59vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-59vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-59vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-59vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-59vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-59vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.19@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 644a63c3-5286-4e41-97a5-767163059200 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 644a63c3-5286-4e41-97a5-767163059200.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.19@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 405da4da-00ed-42b3-89cb-7a414c928fcc r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 405da4da-00ed-42b3-89cb-7a414c928fcc.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-59vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-59vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 644a63c3-5286-4e41-97a5-767163059200 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 644a63c3-5286-4e41-97a5-767163059200.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.25.20@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 644a63c3-5286-4e41-97a5-767163059200 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9372:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.25.19@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9372:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9573:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9573:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9573 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.255@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 0b264490-59a8-4b6a-ba1d-e06db0bb3b7d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 0b264490-59a8-4b6a-ba1d-e06db0bb3b7d.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.255@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem aa1bcf6d-bb69-4a52-b9db-cc8236cd4fed r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem aa1bcf6d-bb69-4a52-b9db-cc8236cd4fed.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-58vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 0b264490-59a8-4b6a-ba1d-e06db0bb3b7d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 0b264490-59a8-4b6a-ba1d-e06db0bb3b7d.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.25.0@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 0b264490-59a8-4b6a-ba1d-e06db0bb3b7d r/w with ordered data mode. Quota mode: journalled.
Lustre: 9379:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.24.255@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9379:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9531:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9531:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9531 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.145@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 6f4f691c-316e-4315-b9f0-d706800f4120 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 6f4f691c-316e-4315-b9f0-d706800f4120.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.145@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 33202d5a-ee17-4ee3-aa4b-00e41be2659a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 33202d5a-ee17-4ee3-aa4b-00e41be2659a.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-91vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 6f4f691c-316e-4315-b9f0-d706800f4120 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 6f4f691c-316e-4315-b9f0-d706800f4120.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.43.146@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 6f4f691c-316e-4315-b9f0-d706800f4120 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9358:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.43.145@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9358:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9552:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9552:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9552 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-69vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-69vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-69vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-69vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-69vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-69vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-69vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-69vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.217@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem d694a1f3-ec79-4cae-a5e9-3a96b9e8a983 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem d694a1f3-ec79-4cae-a5e9-3a96b9e8a983.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.217@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 54cfec91-2fc8-48cc-bbd8-9894ecfbe315 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 54cfec91-2fc8-48cc-bbd8-9894ecfbe315.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-69vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-69vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem d694a1f3-ec79-4cae-a5e9-3a96b9e8a983 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem d694a1f3-ec79-4cae-a5e9-3a96b9e8a983.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.41.218@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem d694a1f3-ec79-4cae-a5e9-3a96b9e8a983 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9358:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.41.217@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9358:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9532:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9532:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9532 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.165@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem b99e739f-8c2f-4535-9450-962654358512 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem b99e739f-8c2f-4535-9450-962654358512.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.165@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 91c7c20b-a689-4cc9-ba7b-1ed7fbdcec2b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 91c7c20b-a689-4cc9-ba7b-1ed7fbdcec2b.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-92vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem b99e739f-8c2f-4535-9450-962654358512 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem b99e739f-8c2f-4535-9450-962654358512.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g2f922b1
LNet: Added LNI 10.240.43.166@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem b99e739f-8c2f-4535-9450-962654358512 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9360:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.43.165@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9360:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9571:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9571:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9571 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-56vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.6@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem f59a0a35-9107-4e3d-af6a-e3205a69d828 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem f59a0a35-9107-4e3d-af6a-e3205a69d828.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.6@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 899e5ac3-65bc-4fbc-a4db-da799c601fd7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 899e5ac3-65bc-4fbc-a4db-da799c601fd7.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-71vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem f59a0a35-9107-4e3d-af6a-e3205a69d828 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem f59a0a35-9107-4e3d-af6a-e3205a69d828.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.26.7@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem f59a0a35-9107-4e3d-af6a-e3205a69d828 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9377:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.26.6@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9377:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9558:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9558:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9558 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-64vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-64vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-56vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-64vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-56vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-64vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-64vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-64vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.119@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 2d53c17f-5a00-496a-83b2-672afebaae0a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 2d53c17f-5a00-496a-83b2-672afebaae0a.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.119@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f239da47-1747-45fa-8ec9-63d874ee62de r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f239da47-1747-45fa-8ec9-63d874ee62de.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-64vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-64vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 2d53c17f-5a00-496a-83b2-672afebaae0a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 2d53c17f-5a00-496a-83b2-672afebaae0a.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.25.121@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 2d53c17f-5a00-496a-83b2-672afebaae0a r/w with ordered data mode. Quota mode: journalled.
Lustre: 9364:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.25.119@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9364:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9564:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9564:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9564 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.30@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 7d55e224-7084-4bba-9622-33c8caf46d5c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 7d55e224-7084-4bba-9622-33c8caf46d5c.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.30@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem b2823da5-2691-4dfb-8ad3-7b661ab741db r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem b2823da5-2691-4dfb-8ad3-7b661ab741db.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 7d55e224-7084-4bba-9622-33c8caf46d5c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 7d55e224-7084-4bba-9622-33c8caf46d5c.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.27.32@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 7d55e224-7084-4bba-9622-33c8caf46d5c r/w with ordered data mode. Quota mode: journalled.
Lustre: 9369:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.27.30@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9369:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9559:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9559:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9559 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-37vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-37vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-37vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-37vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-37vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-37vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-37vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-37vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.89@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem eb27f5dc-d545-4372-905e-0785c7e2b1a2 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem eb27f5dc-d545-4372-905e-0785c7e2b1a2.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.89@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem aa9d04c4-c27c-4626-a8c5-073d57f3dea1 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem aa9d04c4-c27c-4626-a8c5-073d57f3dea1.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-37vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-37vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem eb27f5dc-d545-4372-905e-0785c7e2b1a2 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem eb27f5dc-d545-4372-905e-0785c7e2b1a2.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.23.90@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem eb27f5dc-d545-4372-905e-0785c7e2b1a2 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9364:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.23.89@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9364:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9580:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9580:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9580 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-116vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-58vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.255@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 0c49fe55-eff0-4717-b3f7-49d5c0de1d39 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 0c49fe55-eff0-4717-b3f7-49d5c0de1d39.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.255@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f85e6370-d62c-4368-b9bd-4b2e95d12ee7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f85e6370-d62c-4368-b9bd-4b2e95d12ee7.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-58vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-58vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 0c49fe55-eff0-4717-b3f7-49d5c0de1d39 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 0c49fe55-eff0-4717-b3f7-49d5c0de1d39.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.25.0@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 0c49fe55-eff0-4717-b3f7-49d5c0de1d39 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9385:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.24.255@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9385:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9522:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9522:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9522 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm50.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm148.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm148.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm49.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm148.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm148.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm49.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm50.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.154@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 99cb8d95-b6c3-4ffe-b6e0-8deb02fffa84 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 99cb8d95-b6c3-4ffe-b6e0-8deb02fffa84.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.154@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 599ec458-1f28-49cc-9ed5-6e8fc74e3632 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 599ec458-1f28-49cc-9ed5-6e8fc74e3632.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm50.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm50.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 99cb8d95-b6c3-4ffe-b6e0-8deb02fffa84 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 99cb8d95-b6c3-4ffe-b6e0-8deb02fffa84.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.31.252@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 99cb8d95-b6c3-4ffe-b6e0-8deb02fffa84 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9363:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.31.154@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9363:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9583:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9583:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9583 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm145.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm146.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm147.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm146.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm145.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm147.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm147.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm147.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.250@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 34e5d25c-8d8d-4636-947a-cc2635a50245 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 34e5d25c-8d8d-4636-947a-cc2635a50245.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.250@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 81bfa3e8-b380-473c-abc5-031663c53909 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 81bfa3e8-b380-473c-abc5-031663c53909.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm146.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm146.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 34e5d25c-8d8d-4636-947a-cc2635a50245 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 34e5d25c-8d8d-4636-947a-cc2635a50245.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.31.251@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 34e5d25c-8d8d-4636-947a-cc2635a50245 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9407:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.31.250@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9407:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
LustreError: lustre-MDT0001: not available for connect from 10.240.31.250@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9569:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9569:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9569 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-65vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-65vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-65vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-65vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-65vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-65vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-65vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-65vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.139@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 0185568e-8a41-459a-bfa8-9373a730dcd3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 0185568e-8a41-459a-bfa8-9373a730dcd3.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.139@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem b23f9cdc-e530-479f-b771-30da9aa6b27b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem b23f9cdc-e530-479f-b771-30da9aa6b27b.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-65vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-65vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 0185568e-8a41-459a-bfa8-9373a730dcd3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 0185568e-8a41-459a-bfa8-9373a730dcd3.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.25.140@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 0185568e-8a41-459a-bfa8-9373a730dcd3 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9375:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.25.139@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9375:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9567:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9567:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9567 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-25vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-25vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-25vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-25vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-25vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-25vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-25vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-25vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.111@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 19b2dbf8-4492-493d-9cec-f8e66b4457e6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 19b2dbf8-4492-493d-9cec-f8e66b4457e6.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.111@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem e7c9e41d-15f3-4f27-8600-471f95485beb r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem e7c9e41d-15f3-4f27-8600-471f95485beb.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-25vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-25vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 19b2dbf8-4492-493d-9cec-f8e66b4457e6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 19b2dbf8-4492-493d-9cec-f8e66b4457e6.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.22.112@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 19b2dbf8-4492-493d-9cec-f8e66b4457e6 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9372:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.22.111@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9372:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10079:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10079:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 10079 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm144.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm143.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm143.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm144.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.46@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 883383c4-78e0-4f08-8836-cc717dd635e7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 883383c4-78e0-4f08-8836-cc717dd635e7.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.46@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 57bbbe52-b90d-4b33-b6e5-901df83e7035 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 57bbbe52-b90d-4b33-b6e5-901df83e7035.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.46.47@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm144.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-155vm144.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 883383c4-78e0-4f08-8836-cc717dd635e7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 883383c4-78e0-4f08-8836-cc717dd635e7.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 883383c4-78e0-4f08-8836-cc717dd635e7 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9965:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.46.46@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9965:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9565:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9565:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9565 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-106vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-106vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.194@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 02e3940d-22d1-4b5c-8d68-3f9a2d16d266 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 02e3940d-22d1-4b5c-8d68-3f9a2d16d266.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.194@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 1ddd8889-a998-477d-8356-161ff4a45285 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 1ddd8889-a998-477d-8356-161ff4a45285.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-106vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 02e3940d-22d1-4b5c-8d68-3f9a2d16d266 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 02e3940d-22d1-4b5c-8d68-3f9a2d16d266.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.28.195@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 02e3940d-22d1-4b5c-8d68-3f9a2d16d266 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9371:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.28.194@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9371:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 11331:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 11331:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 11331 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm149.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm148.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm149.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm148.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm149.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm148.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm147.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm148.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm149.trevis.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: trevis-155vm147.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
LNet: Added LNI fd33:3981:3213:f020:0:5254:e:2482@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f020:0:5254:89:aacf@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-0): mounted filesystem 9adb5b23-ff28-4b49-97c2-6baef42106b7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 9adb5b23-ff28-4b49-97c2-6baef42106b7.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=fd33:3981:3213:f020:0:5254:89:aacf@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_R
LDISKFS-fs (dm-1): mounted filesystem 0fe818a5-2613-48c2-a023-19a4b416fed8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 0fe818a5-2613-48c2-a023-19a4b416fed8.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm149.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm148.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm147.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm149.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm148.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm150.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-155vm147.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm149.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-155vm149.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 9adb5b23-ff28-4b49-97c2-6baef42106b7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 9adb5b23-ff28-4b49-97c2-6baef42106b7.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 9adb5b23-ff28-4b49-97c2-6baef42106b7 r/w with ordered data mode. Quota mode: journalled.
Lustre: 11297:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGCfd33:3981:3213:f020:0:5254:89:aacf@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 11297:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10092:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10092:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 10092 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm143.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm144.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm145.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm143.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm144.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.246@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem a3221c37-cb95-4bfb-a7da-196b0b82701c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem a3221c37-cb95-4bfb-a7da-196b0b82701c.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.246@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 08b7c4de-bfe5-4f3e-90ef-982d44211a76 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 08b7c4de-bfe5-4f3e-90ef-982d44211a76.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.46.247@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm144.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm144.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem a3221c37-cb95-4bfb-a7da-196b0b82701c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem a3221c37-cb95-4bfb-a7da-196b0b82701c.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem a3221c37-cb95-4bfb-a7da-196b0b82701c r/w with ordered data mode. Quota mode: journalled.
Lustre: 9972:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.46.246@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9972:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9515:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9515:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9515 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm126.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-155vm126.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.29@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 7509c76d-af12-40d1-867d-af9c67d580b9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 7509c76d-af12-40d1-867d-af9c67d580b9.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.29@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 5b94e373-2e49-48b7-beee-82a743b20ae9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 5b94e373-2e49-48b7-beee-82a743b20ae9.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-155vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 7509c76d-af12-40d1-867d-af9c67d580b9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 7509c76d-af12-40d1-867d-af9c67d580b9.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.46.30@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 7509c76d-af12-40d1-867d-af9c67d580b9 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9363:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.46.29@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9363:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9521:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9521:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9521 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm126.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm126.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.229@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem d7b3f8a5-8658-4858-8b1e-5c8015f54c8f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem d7b3f8a5-8658-4858-8b1e-5c8015f54c8f.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.229@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 5846b675-5b66-4294-8d7f-2db0a994b0f7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 5846b675-5b66-4294-8d7f-2db0a994b0f7.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem d7b3f8a5-8658-4858-8b1e-5c8015f54c8f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem d7b3f8a5-8658-4858-8b1e-5c8015f54c8f.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.46.230@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem d7b3f8a5-8658-4858-8b1e-5c8015f54c8f r/w with ordered data mode. Quota mode: journalled.
Lustre: 9360:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.46.229@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9360:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9492:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9492:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9492 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm122.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm121.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm123.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm123.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm122.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm123.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm121.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm123.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.224@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem ee41b527-1b7d-4851-885b-9a0d836759a6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem ee41b527-1b7d-4851-885b-9a0d836759a6.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.224@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem e8d27033-9cdb-4ba2-a834-e9c94a479b36 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem e8d27033-9cdb-4ba2-a834-e9c94a479b36.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm122.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm122.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem ee41b527-1b7d-4851-885b-9a0d836759a6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ee41b527-1b7d-4851-885b-9a0d836759a6.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_7_g18a1b86
LNet: Added LNI 10.240.46.225@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem ee41b527-1b7d-4851-885b-9a0d836759a6 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9348:0:(mgc_request_server.c:560:mgc_llog_local_copy()) MGC10.240.46.224@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: 9348:0:(tgt_mount.c:146:server_get_mount()) Can't find mount for MGS
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9563:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9563:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9563 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.159@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 1173215e-d015-4fdc-a80a-03cdb009ff4d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 1173215e-d015-4fdc-a80a-03cdb009ff4d.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.159@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 5100809f-41fa-4708-8652-13d0f4edfdf3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 5100809f-41fa-4708-8652-13d0f4edfdf3.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-66vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 1173215e-d015-4fdc-a80a-03cdb009ff4d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 1173215e-d015-4fdc-a80a-03cdb009ff4d.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_6_g55f3447
LNet: Added LNI 10.240.25.160@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 1173215e-d015-4fdc-a80a-03cdb009ff4d r/w with ordered data mode. Quota mode: journalled.
Lustre: 9369:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.25.159@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9546:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9546:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9546 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 593cd2d6-bb34-4fec-b622-071481bb743a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 593cd2d6-bb34-4fec-b622-071481bb743a.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 95779bf9-a23c-4213-83bc-1ce515b877d8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 95779bf9-a23c-4213-83bc-1ce515b877d8.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 593cd2d6-bb34-4fec-b622-071481bb743a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 593cd2d6-bb34-4fec-b622-071481bb743a.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_6_g55f3447
LNet: Added LNI 10.240.22.192@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 593cd2d6-bb34-4fec-b622-071481bb743a r/w with ordered data mode. Quota mode: journalled.
Lustre: 9374:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.22.189@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10216:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10216:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 10216 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-78vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-78vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.147@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 358c5f7e-18d9-412e-b633-26338c64aa86 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 358c5f7e-18d9-412e-b633-26338c64aa86.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.147@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 7984d88a-7821-416a-b586-b1bdfd476951 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 7984d88a-7821-416a-b586-b1bdfd476951.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_25292.104)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.26.149@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-78vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 358c5f7e-18d9-412e-b633-26338c64aa86 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 358c5f7e-18d9-412e-b633-26338c64aa86.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 358c5f7e-18d9-412e-b633-26338c64aa86 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10040:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.26.147@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9600:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9600:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 9600 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-119vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-119vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-119vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-119vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-119vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-119vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-119vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-119vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.193@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem fe854f05-0c30-472a-89b2-2e58b9bfb3f2 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem fe854f05-0c30-472a-89b2-2e58b9bfb3f2.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.193@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 66e68898-9bc0-4892-9b23-1bf55f86652f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 66e68898-9bc0-4892-9b23-1bf55f86652f.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-119vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-119vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem fe854f05-0c30-472a-89b2-2e58b9bfb3f2 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem fe854f05-0c30-472a-89b2-2e58b9bfb3f2.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.29.194@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem fe854f05-0c30-472a-89b2-2e58b9bfb3f2 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9404:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.29.193@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10216:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10216:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 10216 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-78vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.141@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 9280f9ff-a3e8-4fc9-bdf8-cf2f4fe9af88 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 9280f9ff-a3e8-4fc9-bdf8-cf2f4fe9af88.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.141@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 961d166b-b38c-42db-805d-84a2ad21c8d6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 961d166b-b38c-42db-805d-84a2ad21c8d6.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_25293.105)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_6_g55f3447
LNet: Added LNI 10.240.42.142@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-78vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-78vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 9280f9ff-a3e8-4fc9-bdf8-cf2f4fe9af88 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 9280f9ff-a3e8-4fc9-bdf8-cf2f4fe9af88.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 9280f9ff-a3e8-4fc9-bdf8-cf2f4fe9af88 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10040:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.42.141@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9498:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9498:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 9498 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? wakeup_preempt+0x5a/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xa7/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd30/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm42.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm42.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm41.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm40.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm42.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm42.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm40.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm41.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.145@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem ac1fed6e-300f-45d4-9e64-2f138ed1d07d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem ac1fed6e-300f-45d4-9e64-2f138ed1d07d.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.145@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2097152 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 59bc3d49-8841-49a3-882d-609f999a7c16 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 59bc3d49-8841-49a3-882d-609f999a7c16.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm41.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm41.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 4194304 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem ac1fed6e-300f-45d4-9e64-2f138ed1d07d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ac1fed6e-300f-45d4-9e64-2f138ed1d07d.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.31.146@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem ac1fed6e-300f-45d4-9e64-2f138ed1d07d r/w with ordered data mode. Quota mode: journalled.
Lustre: 9351:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.31.145@tcp: no remote llog for lustre-sptlrpc, check MGS config
LustreError: lustre-MDT0001: not available for connect from 10.240.31.145@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9567:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9567:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 1 PID: 9567 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? ttwu_queue_wakelist+0xf2/0x110
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm9.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.106@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 503d87c3-7bfc-4684-9198-a45af29d19d7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 503d87c3-7bfc-4684-9198-a45af29d19d7.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.106@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem d59f88af-6c93-4cbc-867e-b6cd68647eea r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem d59f88af-6c93-4cbc-867e-b6cd68647eea.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 503d87c3-7bfc-4684-9198-a45af29d19d7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 503d87c3-7bfc-4684-9198-a45af29d19d7.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.38.108@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 503d87c3-7bfc-4684-9198-a45af29d19d7 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9372:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.38.106@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9566:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9566:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 1 PID: 9566 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm1.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-24vm1.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-24vm1.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-24vm1.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.153@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 32290e96-4ab0-41f9-87cf-5a481d257f02 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 32290e96-4ab0-41f9-87cf-5a481d257f02.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.153@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem cd5f1eb6-3a8d-4866-91e6-46e4e175d1d4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem cd5f1eb6-3a8d-4866-91e6-46e4e175d1d4.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-53vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 32290e96-4ab0-41f9-87cf-5a481d257f02 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 32290e96-4ab0-41f9-87cf-5a481d257f02.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_6_g55f3447
LNet: Added LNI 10.240.38.80@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 32290e96-4ab0-41f9-87cf-5a481d257f02 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9371:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.40.153@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9554:0:(lraft_lib.c:806:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9554:0:(lraft_lib.c:806:lraft_init()) LBUG
CPU: 0 PID: 9554 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe9d/0xeb0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-37vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-37vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-37vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-37vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-37vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-37vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-37vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-37vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.39.89@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 3a9a8a43-f684-40ac-854a-cc60f76d81dc r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 3a9a8a43-f684-40ac-854a-cc60f76d81dc.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.39.89@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem b02db76d-fcc0-44d7-be50-272b3edaa41d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem b02db76d-fcc0-44d7-be50-272b3edaa41d.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-37vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-37vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 3a9a8a43-f684-40ac-854a-cc60f76d81dc r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 3a9a8a43-f684-40ac-854a-cc60f76d81dc.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_6_g55f3447
LNet: Added LNI 10.240.39.90@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 3a9a8a43-f684-40ac-854a-cc60f76d81dc r/w with ordered data mode. Quota mode: journalled.
Lustre: 9360:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.39.89@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10162:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10162:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 10162 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-55vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-55vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-55vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-67vm4.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-55vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-55vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-55vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-67vm4.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.190@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 017a5676-1141-42d9-a8fb-e252bf9d5b9c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 017a5676-1141-42d9-a8fb-e252bf9d5b9c.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.190@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 6cfb2119-7a3a-4ea8-ac14-a3a543a9e94a r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 6cfb2119-7a3a-4ea8-ac14-a3a543a9e94a.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_25292.105)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.40.192@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-55vm3.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-55vm3.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 017a5676-1141-42d9-a8fb-e252bf9d5b9c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 017a5676-1141-42d9-a8fb-e252bf9d5b9c.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 017a5676-1141-42d9-a8fb-e252bf9d5b9c r/w with ordered data mode. Quota mode: journalled.
Lustre: 10025:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.40.190@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9559:0:(lraft_lib.c:625:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9559:0:(lraft_lib.c:625:lraft_init()) LBUG
CPU: 0 PID: 9559 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xe96/0xea0 [lraft]
? wakeup_preempt+0x5d/0x70
mdt_init0+0xfca/0x1190 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.201@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem a739f14a-1030-45f8-8a01-d3238e4ab199 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem a739f14a-1030-45f8-8a01-d3238e4ab199.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.201@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 45ed46e1-1ad4-4f73-9dd9-9dc150c5aa46 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 45ed46e1-1ad4-4f73-9dd9-9dc150c5aa46.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-81vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem a739f14a-1030-45f8-8a01-d3238e4ab199 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem a739f14a-1030-45f8-8a01-d3238e4ab199.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_5_g75b5a1d
LNet: Added LNI 10.240.42.202@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem a739f14a-1030-45f8-8a01-d3238e4ab199 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9365:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.42.201@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9535:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9535:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9535 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem ad1ed85b-2bfe-4dae-be84-ec4134d69a21 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem ad1ed85b-2bfe-4dae-be84-ec4134d69a21.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 19645edb-b3f9-4511-bddd-6079ead1d7df r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 19645edb-b3f9-4511-bddd-6079ead1d7df.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem ad1ed85b-2bfe-4dae-be84-ec4134d69a21 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ad1ed85b-2bfe-4dae-be84-ec4134d69a21.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.22.190@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem ad1ed85b-2bfe-4dae-be84-ec4134d69a21 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9379:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.22.189@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: Not available for connect from 10.240.22.189@tcp (not set up)
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9550:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9550:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9550 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-138vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem dd954148-4612-4118-9a7f-1a85c90ccdd8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem dd954148-4612-4118-9a7f-1a85c90ccdd8.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.189@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem a0fb8f69-f4f9-4cdd-8ed7-4fab21ad3ed9 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem a0fb8f69-f4f9-4cdd-8ed7-4fab21ad3ed9.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-138vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem dd954148-4612-4118-9a7f-1a85c90ccdd8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem dd954148-4612-4118-9a7f-1a85c90ccdd8.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.22.190@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem dd954148-4612-4118-9a7f-1a85c90ccdd8 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9377:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.22.189@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9570:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9570:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 0 PID: 9570 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-151vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-151vm2.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-151vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-151vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-151vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-151vm2.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-151vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-151vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.90@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 1fa44e27-b71c-4cf1-86d2-f9063e682172 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 1fa44e27-b71c-4cf1-86d2-f9063e682172.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.90@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 354f23d8-9e63-494a-8663-8f2252afda41 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 354f23d8-9e63-494a-8663-8f2252afda41.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-151vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-151vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 1fa44e27-b71c-4cf1-86d2-f9063e682172 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 1fa44e27-b71c-4cf1-86d2-f9063e682172.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.27.91@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 1fa44e27-b71c-4cf1-86d2-f9063e682172 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9375:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.27.90@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: Not available for connect from 10.240.27.90@tcp (not set up)
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9574:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9574:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9574 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.186@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem ca640cd2-aaeb-43c5-897b-097fae94bc12 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem ca640cd2-aaeb-43c5-897b-097fae94bc12.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.186@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 3f002279-a3bd-4e3a-b8fc-02059c273f62 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 3f002279-a3bd-4e3a-b8fc-02059c273f62.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem ca640cd2-aaeb-43c5-897b-097fae94bc12 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ca640cd2-aaeb-43c5-897b-097fae94bc12.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.26.187@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem ca640cd2-aaeb-43c5-897b-097fae94bc12 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9380:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.26.186@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9554:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9554:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9554 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-146vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.29@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem e137351c-0714-468f-b354-6b7e3baa9a9c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem e137351c-0714-468f-b354-6b7e3baa9a9c.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.29@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f8b3ef26-f582-41e0-86ea-37079d8fc7bf r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f8b3ef26-f582-41e0-86ea-37079d8fc7bf.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem e137351c-0714-468f-b354-6b7e3baa9a9c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem e137351c-0714-468f-b354-6b7e3baa9a9c.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.27.30@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem e137351c-0714-468f-b354-6b7e3baa9a9c r/w with ordered data mode. Quota mode: journalled.
Lustre: 9389:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.27.29@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9564:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9564:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9564 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-76vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-76vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-76vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-76vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-76vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-76vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-76vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-76vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.106@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem fd931442-44d8-476c-8383-db3bf5168303 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem fd931442-44d8-476c-8383-db3bf5168303.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.106@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 126adddc-97e2-4e72-a317-f77e50019644 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 126adddc-97e2-4e72-a317-f77e50019644.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-76vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-76vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem fd931442-44d8-476c-8383-db3bf5168303 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem fd931442-44d8-476c-8383-db3bf5168303.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.26.107@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem fd931442-44d8-476c-8383-db3bf5168303 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9373:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.26.106@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10271:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10271:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 10271 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm5.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-107vm4.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.207@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 165bd7d8-3e76-48e7-bc42-6accad18cc78 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 165bd7d8-3e76-48e7-bc42-6accad18cc78.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.28.207@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 6f641238-d901-4265-a673-551e2245769e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 6f641238-d901-4265-a673-551e2245769e.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_24945.105)
Autotest: Test running for 10 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_24945.105)
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.28.208@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-107vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-107vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 165bd7d8-3e76-48e7-bc42-6accad18cc78 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 165bd7d8-3e76-48e7-bc42-6accad18cc78.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 165bd7d8-3e76-48e7-bc42-6accad18cc78 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10132:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.28.207@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10201:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10201:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 1 PID: 10201 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-35vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-35vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-35vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-35vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-35vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-35vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-35vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-35vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.49@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 5843178c-b731-4e60-9629-2995dd2ce34b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 5843178c-b731-4e60-9629-2995dd2ce34b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.49@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem c8f04c3f-4424-432d-ab97-aa53e98ef484 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem c8f04c3f-4424-432d-ab97-aa53e98ef484.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_24944.104)
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.23.50@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-35vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-35vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 5843178c-b731-4e60-9629-2995dd2ce34b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 5843178c-b731-4e60-9629-2995dd2ce34b.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 5843178c-b731-4e60-9629-2995dd2ce34b r/w with ordered data mode. Quota mode: journalled.
Lustre: 10025:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.23.49@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10191:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10191:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 10191 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-53vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.153@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem d5f688f2-cb9a-42ce-bb96-ba62b982ddcd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem d5f688f2-cb9a-42ce-bb96-ba62b982ddcd.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.153@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 972fedc1-ac18-461c-bfe0-fd035aaced52 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 972fedc1-ac18-461c-bfe0-fd035aaced52.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-2_24944.105)
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.40.154@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-53vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-53vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem d5f688f2-cb9a-42ce-bb96-ba62b982ddcd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem d5f688f2-cb9a-42ce-bb96-ba62b982ddcd.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem d5f688f2-cb9a-42ce-bb96-ba62b982ddcd r/w with ordered data mode. Quota mode: journalled.
Lustre: 10015:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.40.153@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9550:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9550:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9550 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-51vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-51vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-51vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-51vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-51vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-51vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-51vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-51vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.113@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem c796b961-877d-4ded-8070-9f4669fadfa1 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem c796b961-877d-4ded-8070-9f4669fadfa1.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.113@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 8b06e7ca-fdd9-45e3-a3b7-0392eb2da748 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 8b06e7ca-fdd9-45e3-a3b7-0392eb2da748.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-51vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-51vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem c796b961-877d-4ded-8070-9f4669fadfa1 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem c796b961-877d-4ded-8070-9f4669fadfa1.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.40.114@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem c796b961-877d-4ded-8070-9f4669fadfa1 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9356:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.40.113@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9554:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9554:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 1 PID: 9554 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-77vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-77vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-77vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-77vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.126@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem c8a36799-8e95-49e6-8b04-75d2205ad82b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem c8a36799-8e95-49e6-8b04-75d2205ad82b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.126@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem aeb24b09-59e1-419f-8d8f-0d6ed73389dd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem aeb24b09-59e1-419f-8d8f-0d6ed73389dd.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-77vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem c8a36799-8e95-49e6-8b04-75d2205ad82b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem c8a36799-8e95-49e6-8b04-75d2205ad82b.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.26.127@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem c8a36799-8e95-49e6-8b04-75d2205ad82b r/w with ordered data mode. Quota mode: journalled.
Lustre: 9360:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.26.126@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9561:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9561:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 1 PID: 9561 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-43vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-43vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-43vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-43vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.209@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem e933ff88-c80b-49f1-83b8-089562be104f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem e933ff88-c80b-49f1-83b8-089562be104f.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.23.209@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f1fd42a7-c8b0-4b74-9a25-cd0a7084c34e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f1fd42a7-c8b0-4b74-9a25-cd0a7084c34e.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-43vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem e933ff88-c80b-49f1-83b8-089562be104f r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem e933ff88-c80b-49f1-83b8-089562be104f.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.23.210@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem e933ff88-c80b-49f1-83b8-089562be104f r/w with ordered data mode. Quota mode: journalled.
Lustre: 9367:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.23.209@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9556:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9556:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9556 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-30vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-30vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-30vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-30vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-30vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-30vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-30vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-30vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.205@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 89e15d65-5e19-45ff-afbf-92d75af4ae90 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 89e15d65-5e19-45ff-afbf-92d75af4ae90.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.205@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem f20becbb-7c50-485e-ba5d-2e0ebca69b46 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem f20becbb-7c50-485e-ba5d-2e0ebca69b46.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-30vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-30vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 89e15d65-5e19-45ff-afbf-92d75af4ae90 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 89e15d65-5e19-45ff-afbf-92d75af4ae90.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.38.206@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 89e15d65-5e19-45ff-afbf-92d75af4ae90 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9361:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.38.205@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: Not available for connect from 10.240.38.205@tcp (not set up)
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9542:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9542:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 1 PID: 9542 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm8.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-66vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.159@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 3c6fd666-e630-4e43-b22c-db02ea5fb208 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 3c6fd666-e630-4e43-b22c-db02ea5fb208.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.159@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem b57596f7-f50a-4c1b-81fc-daa2904e42e6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem b57596f7-f50a-4c1b-81fc-daa2904e42e6.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-66vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-66vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 3c6fd666-e630-4e43-b22c-db02ea5fb208 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 3c6fd666-e630-4e43-b22c-db02ea5fb208.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.25.160@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 3c6fd666-e630-4e43-b22c-db02ea5fb208 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9348:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.25.159@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9560:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9560:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 1 PID: 9560 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-31vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-31vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-31vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-31vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-31vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-31vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-31vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-31vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.225@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem d8c62f0d-1583-42fb-86e6-0d1d35c5b084 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem d8c62f0d-1583-42fb-86e6-0d1d35c5b084.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.225@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem d953230c-e1aa-4e22-bcd4-4e49afeb3a7e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem d953230c-e1aa-4e22-bcd4-4e49afeb3a7e.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-31vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-31vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem d8c62f0d-1583-42fb-86e6-0d1d35c5b084 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem d8c62f0d-1583-42fb-86e6-0d1d35c5b084.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.38.226@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem d8c62f0d-1583-42fb-86e6-0d1d35c5b084 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9365:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.38.225@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9537:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9537:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9537 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-70vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-70vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-70vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-70vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-70vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-70vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-70vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-70vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.237@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 7f2e9447-f7c3-461a-9a61-0ab66fc09035 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 7f2e9447-f7c3-461a-9a61-0ab66fc09035.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.237@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 4318506a-fc35-4c49-8d72-f2723a2b3fc0 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 4318506a-fc35-4c49-8d72-f2723a2b3fc0.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-70vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-70vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 7f2e9447-f7c3-461a-9a61-0ab66fc09035 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 7f2e9447-f7c3-461a-9a61-0ab66fc09035.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.41.238@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 7f2e9447-f7c3-461a-9a61-0ab66fc09035 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9343:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.41.237@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9559:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9559:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9559 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm12.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-80vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.186@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem dc062bec-a016-4597-b29f-fa841c846bd0 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem dc062bec-a016-4597-b29f-fa841c846bd0.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.186@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 171709e7-8910-4aa0-9c3d-46e01b3e24c4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 171709e7-8910-4aa0-9c3d-46e01b3e24c4.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem dc062bec-a016-4597-b29f-fa841c846bd0 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem dc062bec-a016-4597-b29f-fa841c846bd0.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.26.187@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem dc062bec-a016-4597-b29f-fa841c846bd0 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9375:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.26.186@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9526:0:(lraft_lib.c:573:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9526:0:(lraft_lib.c:573:lraft_init()) LBUG
CPU: 0 PID: 9526 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-91vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.145@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 5f9b0f67-f686-47dd-8279-eb3e8aa57c69 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 5f9b0f67-f686-47dd-8279-eb3e8aa57c69.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.145@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem dbe9fb48-3a34-4612-bd5a-823f156f0d7c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem dbe9fb48-3a34-4612-bd5a-823f156f0d7c.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-91vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-91vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 5f9b0f67-f686-47dd-8279-eb3e8aa57c69 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 5f9b0f67-f686-47dd-8279-eb3e8aa57c69.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_24_g26f738a
LNet: Added LNI 10.240.43.146@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 5f9b0f67-f686-47dd-8279-eb3e8aa57c69 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9367:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.43.145@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 10251:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 10251:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 0 PID: 10251 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-59vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-59vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-59vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-59vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-59vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-59vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-59vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-59vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.17@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem e64923e3-de5c-4f50-9d81-ab7f2c3b1997 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem e64923e3-de5c-4f50-9d81-ab7f2c3b1997.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.41.17@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 8547c26c-2907-4a98-9336-ef20505b2b33 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 8547c26c-2907-4a98-9336-ef20505b2b33.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Autotest: Test running for 5 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_24945.104)
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.41.18@tcp [8/256/0/180]
LNet: Accept all, port 7988
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-59vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-59vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Autotest: Test running for 10 minutes (lustre-b_es-reviews_review-dne-selinux-ssk-part-1_24945.104)
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem e64923e3-de5c-4f50-9d81-ab7f2c3b1997 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem e64923e3-de5c-4f50-9d81-ab7f2c3b1997.
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem e64923e3-de5c-4f50-9d81-ab7f2c3b1997 r/w with ordered data mode. Quota mode: journalled.
Lustre: 10075:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.41.17@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9577:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9577:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9577 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-153vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-153vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-153vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-153vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-153vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-153vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-153vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-153vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.209@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 31cef3bb-ed34-4100-9374-264def6563a5 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 31cef3bb-ed34-4100-9374-264def6563a5.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.27.209@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 5329c9d3-8274-477a-91fc-a8a216710724 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 5329c9d3-8274-477a-91fc-a8a216710724.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-153vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-153vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 31cef3bb-ed34-4100-9374-264def6563a5 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 31cef3bb-ed34-4100-9374-264def6563a5.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.27.210@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 31cef3bb-ed34-4100-9374-264def6563a5 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9383:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.27.209@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9571:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9571:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9571 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-52vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-52vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-52vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-52vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.133@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 242c1c12-168e-4ca4-aea9-f2db50a01a04 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 242c1c12-168e-4ca4-aea9-f2db50a01a04.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.133@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem ca7ae815-0ae0-486f-aaf4-c5bfeb8503ac r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem ca7ae815-0ae0-486f-aaf4-c5bfeb8503ac.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-52vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 242c1c12-168e-4ca4-aea9-f2db50a01a04 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 242c1c12-168e-4ca4-aea9-f2db50a01a04.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.24.134@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 242c1c12-168e-4ca4-aea9-f2db50a01a04 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9376:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.24.133@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9555:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9555:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9555 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-32vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-32vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-32vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-32vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-32vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-32vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-32vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-32vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.245@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 101dbff8-585a-47aa-8a92-c18445a62e65 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 101dbff8-585a-47aa-8a92-c18445a62e65.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.245@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem d444b9c7-479b-4096-ac49-6c5ae147f9ea r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem d444b9c7-479b-4096-ac49-6c5ae147f9ea.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-32vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-32vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 101dbff8-585a-47aa-8a92-c18445a62e65 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 101dbff8-585a-47aa-8a92-c18445a62e65.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.38.246@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 101dbff8-585a-47aa-8a92-c18445a62e65 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9361:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.38.245@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9561:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9561:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9561 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-57vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-57vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-57vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-57vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-57vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-57vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-57vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-57vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.233@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 59b49d3d-65e9-4bbb-9a84-0a67be18a4f3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 59b49d3d-65e9-4bbb-9a84-0a67be18a4f3.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.233@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1929379 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 8f443afb-3134-4eda-adeb-386dd8b072ea r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 8f443afb-3134-4eda-adeb-386dd8b072ea.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-57vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-57vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 59b49d3d-65e9-4bbb-9a84-0a67be18a4f3 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 59b49d3d-65e9-4bbb-9a84-0a67be18a4f3.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.24.234@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 59b49d3d-65e9-4bbb-9a84-0a67be18a4f3 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9367:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.24.233@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9570:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9570:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 0 PID: 9570 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-55vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-55vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-55vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-55vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-55vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-55vm7.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-55vm6.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-55vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.193@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 0e7d7baf-06af-430a-82a0-851f590c148d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 0e7d7baf-06af-430a-82a0-851f590c148d.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.193@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem ec9677bf-92be-4cb0-9833-1151ccaa73b8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem ec9677bf-92be-4cb0-9833-1151ccaa73b8.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-55vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-55vm6.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 0e7d7baf-06af-430a-82a0-851f590c148d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 0e7d7baf-06af-430a-82a0-851f590c148d.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.24.194@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 0e7d7baf-06af-430a-82a0-851f590c148d r/w with ordered data mode. Quota mode: journalled.
Lustre: 9376:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.24.193@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9546:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9546:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 0 PID: 9546 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-84vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-84vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-84vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-84vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.5@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem d1600f6a-316a-4290-8ecd-412d1030282b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem d1600f6a-316a-4290-8ecd-412d1030282b.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.5@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 23fc9717-c15b-49d7-b08f-7702b9631af8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 23fc9717-c15b-49d7-b08f-7702b9631af8.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-84vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem d1600f6a-316a-4290-8ecd-412d1030282b r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem d1600f6a-316a-4290-8ecd-412d1030282b.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.43.6@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem d1600f6a-316a-4290-8ecd-412d1030282b r/w with ordered data mode. Quota mode: journalled.
Lustre: 9351:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.43.5@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9553:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9553:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 1 PID: 9553 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-83vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-83vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-83vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-83vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-83vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-83vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-83vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-83vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.241@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 23cd94b3-ad80-45c1-b25d-92dc6d1b6111 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 23cd94b3-ad80-45c1-b25d-92dc6d1b6111.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.241@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem b7dd4e4d-c634-475b-8320-c18fc3fa1d0c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem b7dd4e4d-c634-475b-8320-c18fc3fa1d0c.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-83vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-83vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 23cd94b3-ad80-45c1-b25d-92dc6d1b6111 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 23cd94b3-ad80-45c1-b25d-92dc6d1b6111.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.42.242@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 23cd94b3-ad80-45c1-b25d-92dc6d1b6111 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9357:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.42.241@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: Not available for connect from 10.240.42.241@tcp (not set up)
Lustre: lustre-MDT0001: new disk, initializing

LustreError: 9546:0:(lraft_lib.c:685:lraft_init()) ASSERTION( lraft_target_reg_cb ) failed:
LustreError: 9546:0:(lraft_lib.c:685:lraft_init()) LBUG
CPU: 0 PID: 9546 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lraft_init+0xded/0xe00 [lraft]
mdt_init0+0xfac/0x1170 [mdt]
mdt_device_alloc+0x68/0x180 [mdt]
obd_setup+0xaa/0x300 [obdclass]
class_setup+0x5cb/0x800 [obdclass]
class_process_config+0x173c/0x1bc0 [obdclass]
? class_config_llog_handler+0x620/0x1380 [obdclass]
class_config_llog_handler+0x79d/0x1380 [obdclass]
llog_process_thread+0xd33/0x1910 [obdclass]
? __pfx_llog_process_thread_daemonize+0x10/0x10 [obdclass]
llog_process_thread_daemonize+0x99/0xe0 [obdclass]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-82vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-82vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-82vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-82vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-82vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-82vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-82vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-82vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.221@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem 5848b1bd-ed04-4d16-bf63-5fb205a89ff4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 5848b1bd-ed04-4d16-bf63-5fb205a89ff4.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.221@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=1981808 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 81faad16-4d38-43ab-9295-48fa81b172b0 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 81faad16-4d38-43ab-9295-48fa81b172b0.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-82vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-82vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 3964928 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 5848b1bd-ed04-4d16-bf63-5fb205a89ff4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 5848b1bd-ed04-4d16-bf63-5fb205a89ff4.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn18_25_g8e13084
LNet: Added LNI 10.240.42.222@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 5848b1bd-ed04-4d16-bf63-5fb205a89ff4 r/w with ordered data mode. Quota mode: journalled.
Lustre: 9359:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.42.221@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing