Edit new crash report

Reason	Crashing Function	Where to cut Backtrace	Reports Count
general protection fault,	raft_node_is_active	raft_get_num_voting_nodes lraft_timerd kthread ret_from_fork	14

Failing Test	Full Crash	Messages before crash	Link and date
sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1)	general protection fault, probably for non-canonical address 0xca2c8ecb4093b597: 0000 [#1] PREEMPT SMP PTI CPU: 1 UID: 0 PID: 1143263 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffae4d062cfe90 EFLAGS: 00010202 RAX: ffff9b02a3f2ad90 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ffff9b029cf1c140 RSI: 0000000000000000 RDI: ca2c8ecb4093b57f RBP: ffff9b02e84d3c00 R08: 0000000000000040 R09: 0000000000000002 R10: 0000000000000000 R11: ffff9b0288f28090 R12: 0000000000000001 R13: 0000128489363d72 R14: ffff9b02b4b89000 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ffff9b02fbd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffc1b8ebe18 CR3: 000000012b0ea004 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xd2/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs mlx5_ib ib_uverbs macsec rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel mlx5_core kvm sunrpc rapl iTCO_wdt iTCO_vendor_support mlxfw psample tls pcspkr virtio_net i2c_i801 lpc_ich i2c_smbus pci_hyperv_intf virtio_balloon net_failover failover joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core [last unloaded: dm_flakey] Crash core archive creation started: kernel-crash.onyx-146vm8.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-146vm8.vmcore.tar.gz	Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-143vm34.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-143vm34.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl \| grep ' IN osc ' 2>/dev/null \| wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod..mdt_hash=crush Lustre: 1127397:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/lod..mdt_hash=crush (mode = 0) failed: rc = -17 Lustre: 1127397:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Autotest: Test running for 335 minutes (lustre-b_es-reviews_review-dne-part-7_31815.121) \| 2026-10-23 13:33:47 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LDISKFS-fs (dm-3): unmounting filesystem 84da455e-b011-40e6-a474-a82effd5ddd2. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 Lustre: Failing over lustre-MDT0002 LDISKFS-fs (dm-4): unmounting filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem 84da455e-b011-40e6-a474-a82effd5ddd2 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ . Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-3): unmounting filesystem 84da455e-b011-40e6-a474-a82effd5ddd2. Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=400000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled. LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488. Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488. Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt LDISKFS-fs (dm-4): mounted filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ . Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-4): unmounting filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e. Lustre: DEBUG MARKER: [ -e /dev/mapper/mds3_flakey ] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.54@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=400000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds3_flakey LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled. LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4. Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4. Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey lustre-MDT0002 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-..auto_scrub=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-..full_scrub_ratio=0 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190 Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004284:0x1:0x0]/40: rc = 0 Lustre: * cfs_fail_loc=190, val=1* Lustre: Skipped 56 previous similar messages Lustre: lustre-MDT0002: trigger OI scrub by RPC for [0x280004281:0x1:0x0]/250 with flags 0x52: rc = 0 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub \| Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub \| Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 Lustre: Failing over lustre-MDT0002 LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub \|	Link to test (2026-10-23 09:46)
conf-sanity test 50c: lazystatfs one server down	general protection fault, probably for non-canonical address 0xfe1c15db5069ac8e: 0000 [#1] PREEMPT SMP NOPTI CPU: 0 UID: 0 PID: 587560 Comm: lraft_0003 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: f0 d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 f4 f0 d1 0f 1f 44 00 RSP: 0018:ff63da388aa2be90 EFLAGS: 00010246 RAX: ff407914c4d96758 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ff407914c4d96110 RSI: 0000000000000000 RDI: fe1c15db5069ac76 RBP: ff407914c2b36200 R08: 0000000000000040 R09: 0000000000000004 R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 00000d156ad7fc04 R14: ff407914cf0af000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ff4079153ba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000564965aad164 CR3: 00000001315d2002 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: lzstd(OE) llz4hc(OE) llz4(OE) obdecho(OE) ec(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support virtio_net net_failover i2c_i801 lpc_ich virtio_balloon failover pcspkr i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_blk serio_raw Crash core archive creation started: kernel-crash.trevis-152vm163.vmcore.tar.gz Crash core archive created: kernel-crash.trevis-152vm163.vmcore.tar.gz	Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem dd31694d-9434-473e-bda8-dde26d69ae88 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 590f1cbc-f830-4898-af83-0b17f3dd0c00 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-10-13 12:59)
sanity-sec test 27ab: test nodemap idmap offset	general protection fault, probably for non-canonical address 0x82c931d33a241cd6: 0000 [#1] PREEMPT SMP PTI CPU: 0 UID: 0 PID: 549916 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffac4c4b563e90 EFLAGS: 00010246 RAX: ffff9ba7c1daff18 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff9ba7c1daf968 RSI: 0000000000000000 RDI: 82c931d33a241cbe RBP: ffff9ba7ea2bc400 R08: 0000000000000040 R09: 000000000000000f R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 00000818885cff29 R14: ffff9ba7e3654000 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ffff9ba83bc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff07209e88 CR3: 0000000123464001 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xd2/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs mlx5_ib ib_uverbs macsec rfkill intel_rapl_msr intel_rapl_common mlx5_core intel_uncore_frequency_common kvm_intel sunrpc kvm iTCO_wdt iTCO_vendor_support rapl virtio_net pcspkr net_failover mlxfw psample failover virtio_balloon tls i2c_i801 lpc_ich i2c_smbus pci_hyperv_intf joydev loop fuse dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core Unloaded tainted modules: lzstd(OE):1 llz4hc(OE):1 llz4(OE):1 obdecho(OE):1 libcfs(OE):2 lnet(OE):2 ksocklnd(OE):2 obdclass(OE):2 ldiskfs(OE):2 ptlrpc(OE):2 lquota(OE):2 ptlrpc_gss(OE):2 lraft(OE):2 mgc(OE):2 mgs(OE):2 osc(OE):2 fld(OE):2 lov(OE):2 fid(OE):2 lfsck(OE):2 lod(OE):2 osd_ldiskfs(OE):2 mdt(OE):2 lmv(OE):2 mdc(OE):2 lustre(OE):2 osp(OE):2 mdd(OE):2 [last unloaded: lzstd(OE)] Crash core archive creation started: kernel-crash.onyx-141vm38.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-141vm38.vmcore.tar.gz	Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add Test18109 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add OffsetTest Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_offset --name Test18109 --offset 100000 --limit 200000 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_offset --name OffsetTest --offset 150000 --limit 100000 LustreError: 530213:0:(nodemap_handler.c:5256:nodemap_add_offset()) OffsetTest: new offset 150000+100000 overlaps with existing nodemap Test18109 offset 100000+200000: rc = -34 LustreError: 530213:0:(nodemap_handler.c:6589:server_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE04C for OffsetTest: rc = -34 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name Test18109 --idtype uid --idmap 500-509:0-9 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.idmap Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name Test18109 --range 1.1.1.1@tcp777 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property squash_uid --value 65534 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property squash_gid --value 65534 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 1 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_idmap --name Test18109 --idtype uid --idmap 500-509:0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name Test18109 --idtype uid --idmap 500-509:200000-200010 LustreError: 533554:0:(nodemap_handler.c:753:nodemap_parse_idmap()) Test18109: range length mismatch between client id 500-509 and fs id 200000-200010: rc = -22 LustreError: 533554:0:(nodemap_handler.c:6589:server_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE044 for Test18109: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.idmap Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0 Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: lustre-MDT0000: Not available for connect from 10.240.28.195@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && LustreError: lustre-MDT0000: not available for connect from 10.240.26.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: 514712:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.28.195@tcp arrived at 1791550016 with bad export cookie 1158799906114673421 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.28.195@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 21 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: 514713:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1791550025 with bad export cookie 1158799906114672980 LustreError: 514713:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: MGC10.240.26.66@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 514722:0:(mgc_request.c:616:do_requeue()) failed processing log: -5 Lustre: lustre-MDT0002: Not available for connect from 10.240.26.35@tcp (stopping) Lustre: Skipped 2 previous similar messages LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7db169c00 x1878576097322112/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LDISKFS-fs (dm-4): unmounting filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3 Lustre: 542209:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: lustre-OST0007-osc-MDT0000: operation ost_connect to node 10.240.26.35@tcp failed: rc = -11 LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-180vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl \| grep ' IN osc ' 2>/dev/null \| wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod..mdt_hash=crush Lustre: 547102:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/lod..mdt_hash=crush (mode = 0) failed: rc = -17 Lustre: 547102:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_offset --name Test18109 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7f1d6fb80 x1878576097657728/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.28.195@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 513733:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7db176d80 x1878576097658624/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.28.195@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 513733:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LDISKFS-fs (dm-3): unmounting filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0000: not available for connect from 10.240.28.195@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages LustreError: 537203:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.28.195@tcp arrived at 1791550248 with bad export cookie 1158799906114676410 LustreError: 537203:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.28.195@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: 542245:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1791550257 with bad export cookie 1158799906114675990 LustreError: 542245:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: MGC10.240.26.66@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7da823100 x1878576097670528/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 10.240.28.195@tcp (stopping) Lustre: Skipped 14 previous similar messages LDISKFS-fs (dm-4): unmounting filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Autotest: Test running for 145 minutes (lustre-b_es-reviews_review-dne-exa7-part-1_31463.223) \| 2026-10-09 12:51:13 Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 33 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all	Link to test (2026-10-09 11:43)
sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases	general protection fault, probably for non-canonical address 0x9cb60657e6389348: 0000 [#1] SMP NOPTI CPU: 1 UID: 0 PID: 1048661 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: e7 f8 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 74 e7 f8 0f 1f 44 00 RSP: 0018:ff664d7a4d55fe78 EFLAGS: 00010202 RAX: ff3515cca7a59bc0 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ff3515cca624b2e8 RSI: 0000000000000000 RDI: 9cb60657e6389330 RBP: ff3515ccc03d1400 R08: 0000000000000040 R09: 000000000000000f R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000001 R13: 00000f9238e69bb3 R14: ffffffffc1e93050 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ff3515ccfbb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffcc7fa7578 CR3: 0000000137f5c006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? srso_alias_return_thunk+0x5/0xfbef5 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] ? srso_alias_return_thunk+0x5/0xfbef5 raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass pcspkr i2c_i801 lpc_ich i2c_smbus virtio_balloon joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata virtio_net crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel net_failover failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink [last unloaded: dm_flakey] Crash core archive creation started: kernel-crash.onyx-156vm44.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-156vm44.vmcore.tar.gz	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 Lustre: Failing over lustre-MDT0002 LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: invalid oi count 63, remove them, then set it to 64 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: lustre-MDT0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub \| Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub \| Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub Autotest: Test running for 280 minutes (lustre-b_es-reviews_review-dne-part-7_31396.128) \| 2026-10-05 21:15:11 Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3 Lustre: 1028745:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17 Lustre: 1028745:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl \| grep ' IN osc ' 2>/dev/null \| wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod..mdt_hash=crush Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 LustreError: MGC10.240.28.95@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: Failing over lustre-MDT0002 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: invalid oi count 58, remove them, then set it to 64 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: lustre-MDT0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub \| Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub \| Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl \| grep ' IN osc ' 2>/dev/null \| wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod..mdt_hash=crush Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && LustreError: lustre-MDT0000: not available for connect from 10.240.28.94@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 73 previous similar messages Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages LustreError: 1034605:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.244@tcp arrived at 1791235114 with bad export cookie 10353082178918664466 LustreError: 1034605:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 9 previous similar messages LustreError: lustre-MDT0001-osp-MDT0002: operation raft_appendentries to node 10.240.27.244@tcp failed: rc = -107 LustreError: Skipped 10 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3 Lustre: Failing over lustre-MDT0002 LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196. Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9 Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27 Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9. Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: invalid oi count 56, remove them, then set it to 64 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0000.oi_scrub Lustre: lustre-MDT0000: Recovery over after 0:18, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-.lustre-MDT0002.oi_scrub	Link to test (2026-10-05 17:31)
conf-sanity test 113: Shadow mountpoint correctly report ro/rw for mounts	general protection fault, probably for non-canonical address 0xa0f272be9ded70a0: 0000 [#1] PREEMPT SMP NOPTI CPU: 1 UID: 0 PID: 1532905 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: 31 d4 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 04 31 d4 0f 1f 44 00 RSP: 0018:ff6afccc0e933e90 EFLAGS: 00010202 RAX: ff2ab58904f4c7c0 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ff2ab5892c5e26f8 RSI: 0000000000000000 RDI: a0f272be9ded7088 RBP: ff2ab58936017400 R08: 0000000000000040 R09: 0000000000000004 R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000001 R13: 00001b7bc7467ea5 R14: ff2ab589030d5008 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ff2ab5897bb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055f2437d5048 CR3: 000000011fb3e002 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support pcspkr virtio_net lpc_ich i2c_i801 virtio_balloon net_failover failover i2c_smbus joydev loop fuse dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw Unloaded tainted modules: lzstd(OE):12 llz4hc(OE):12 llz4(OE):12 libcfs(OE):21 lnet(OE):14 ksocklnd(OE):14 obdclass(OE):14 ldiskfs(OE):14 ptlrpc(OE):14 lquota(OE):14 ptlrpc_gss(OE):14 lraft(OE):14 mgc(OE):14 osc(OE):14 fld(OE):14 lov(OE):14 fid(OE):14 lfsck(OE):14 lod(OE):14 osd_ldiskfs(OE):14 mdt(OE):14 lmv(OE):14 mdc(OE):14 lustre(OE):14 osp(OE):14 obdecho(OE):12 mdd(OE):14 [last unloaded: lzstd(OE)] Crash core archive creation started: kernel-crash.onyx-157vm25.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-157vm25.vmcore.tar.gz	Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts); Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem 21718c17-d480-4cef-9115-6dd7dcd008ec r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey;	Link to test (2026-09-26 17:04)
replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1	general protection fault, probably for non-canonical address 0x2c113f106f93cfea: 0000 [#1] SMP NOPTI CPU: 1 UID: 0 PID: 213438 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: 8a dd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 c4 8a dd 0f 1f 44 00 RSP: 0018:ff5293678beefe78 EFLAGS: 00010202 RAX: ff396b5c6cde1b50 RBX: 0000000000000001 RCX: 0000000000000000 RDX: ff396b5c6cc25e38 RSI: 0000000000000000 RDI: 2c113f106f93cfd2 RBP: ff396b5c6d63be00 R08: 0000000000000040 R09: 0000000000000002 R10: 0000000000000000 R11: ff396b5c43980090 R12: 0000000000000001 R13: 000006d9cab2c23a R14: ffffffffc1c5e050 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ff396b5cbbb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055c13b6e1158 CR3: 0000000132aaa006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] ? srso_alias_return_thunk+0x5/0xfbef5 raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass pcspkr i2c_i801 virtio_balloon lpc_ich i2c_smbus joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 virtio_net crct10dif_pclmul ahci crc32_pclmul libahci crc32c_intel ghash_clmulni_intel libata net_failover virtio_blk failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink Crash core archive creation started: kernel-crash.onyx-156vm171.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-156vm171.vmcore.tar.gz	Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 Lustre: * cfs_fail_loc=1701, val=2147483648* LustreError: 28971:0:(ldlm_lib.c:3394:target_send_reply_msg()) @@@ dropping reply req@ff396b5c67d5b100 x1877403953518848/t64424509950(0) o1000->lustre-MDT0000-mdtlov_UUID@10.240.24.18@tcp:497/0 lens 1320/4320 e 0 to 0 dl 1790439207 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up1-0.0' uid:0 gid:0 projid:4294967295 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds2 Lustre: Failing over lustre-MDT0001 LDISKFS-fs (dm-3): unmounting filesystem b7c698fa-8b94-4eaa-bacb-dbd6c9d92458. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 145 previous similar messages Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem b7c698fa-8b94-4eaa-bacb-dbd6c9d92458 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 5 clients reconnect Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-09-26 13:23)
sanity-sec test 62: e2fsck with encrypted files	general protection fault, probably for non-canonical address 0x4e895d66dee964d2: 0000 [#1] SMP NOPTI CPU: 0 UID: 0 PID: 561185 Comm: lraft_0003 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: 74 e2 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 44 74 e2 0f 1f 44 00 RSP: 0018:ff8b8b95cadefe78 EFLAGS: 00010246 RAX: ff48a559100f7ef0 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ff48a559100f7658 RSI: 0000000000000000 RDI: 4e895d66dee964ba RBP: ff48a5590096e000 R08: 0000000000000040 R09: 0000000000000002 R10: 0000000000000000 R11: ff48a5590613c390 R12: 0000000000000000 R13: 00000d8d21c89d76 R14: ffffffffc19c6050 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ff48a5593ba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffdff591a48 CR3: 000000010b1da006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass i2c_i801 pcspkr i2c_smbus virtio_balloon lpc_ich joydev loop fuse vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata virtio_net crct10dif_pclmul crc32_pclmul net_failover crc32c_intel failover ghash_clmulni_intel virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink Unloaded tainted modules: lzstd(OE):6 llz4hc(OE):6 llz4(OE):6 libcfs(OE):7 lnet(OE):7 ksocklnd(OE):7 obdclass(OE):7 ldiskfs(OE):7 ptlrpc(OE):7 lquota(OE):7 ptlrpc_gss(OE):7 lraft(OE):7 mgc(OE):7 osc(OE):7 fld(OE):7 lov(OE):7 fid(OE):7 lfsck(OE):7 lod(OE):7 osd_ldiskfs(OE):7 mdt(OE):7 lmv(OE):7 mdc(OE):7 lustre(OE):7 osp(OE):7 mdd(OE):7 obdecho(OE):7 [last unloaded: lzstd(OE)] Crash core archive creation started: kernel-crash.onyx-156vm121.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-156vm121.vmcore.tar.gz	LustreError: lustre-MDT0000-osp-MDT0003: operation mds_statfs to node 10.240.30.18@tcp failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0003: Connection to lustre-MDT0000 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 10.240.30.18@tcp failed: rc = -107 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2 LustreError: MGC10.240.30.18@tcp: Connection to MGS (at 10.240.30.18@tcp) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001: Not available for connect from 10.240.30.18@tcp (stopping) LustreError: 525337:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff48a558fa5c5f80 x1877192418673920/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.30.18@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295 LustreError: 525337:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: lustre-MDT0001-osp-MDT0003: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 525338:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff48a558fa5c5c00 x1877192418675456/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.30.18@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295 LustreError: 525338:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem 042f26d0-cd46-4437-9b82-841a32be1870. LustreError: lustre-MDT0001: not available for connect from 10.240.30.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0001: not available for connect from 10.240.30.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 LustreError: lustre-MDT0001: not available for connect from 10.240.30.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0003: Not available for connect from 10.240.30.17@tcp (stopping) LDISKFS-fs (dm-4): unmounting filesystem 40b86320-81f7-442c-aae2-8bcdc7f62b06. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts); Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem 042f26d0-cd46-4437-9b82-841a32be1870 r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null LustreError: lustre-MDT0003: not available for connect from 10.240.30.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 40b86320-81f7-442c-aae2-8bcdc7f62b06 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-09-24 04:33)
sanity-sec test 75b: test resource ID repair	general protection fault, probably for non-canonical address 0xeb70f32fdccb6da2: 0000 [#1] SMP PTI CPU: 0 UID: 0 PID: 1102026 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffd27f4a803e78 EFLAGS: 00010246 RAX: ffff8cb2aa1e7c38 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8cb2aa1e7cb8 RSI: 0000000000000000 RDI: eb70f32fdccb6d8a RBP: ffff8cb2a8221600 R08: 0000000000000040 R09: 000000000000000f R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 00000b087f62bc9a R14: ffffffffc17d6050 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ffff8cb2fbc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f5b5cecec40 CR3: 0000000138694003 CR4: 0000000000172ef0 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfd/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass rapl iTCO_wdt iTCO_vendor_support pcspkr virtio_balloon i2c_i801 i2c_smbus lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 virtio_net ahci libahci net_failover libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink Unloaded tainted modules: lzstd(OE):6 llz4hc(OE):6 llz4(OE):6 ptlrpc_gss(OE):6 obdecho(OE):6 libcfs(OE):7 lnet(OE):7 ksocklnd(OE):7 obdclass(OE):7 ldiskfs(OE):7 ptlrpc(OE):7 lquota(OE):7 lraft(OE):7 mgc(OE):7 mgs(OE):7 osc(OE):7 fld(OE):7 lov(OE):7 fid(OE):7 lfsck(OE):7 lod(OE):7 osd_ldiskfs(OE):7 mdt(OE):7 lmv(OE):7 mdc(OE):7 lustre(OE):7 osp(OE):7 mdd(OE):7 [last unloaded: lzstd(OE)] Crash core archive creation started: kernel-crash.onyx-80vm9.vmcore.tar.gz Crash core archive created: kernel-crash.onyx-80vm9.vmcore.tar.gz	Lustre: DEBUG MARKER: lctl set_param -n os[cd].MDT.force_sync=1 Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.osc.old_sync_processed 2> /dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.osc*.old_sync_processed Lustre: DEBUG MARKER: sync; sync; echo 3 > /proc/sys/vm/drop_caches bash (1099989): drop_caches: 3 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 LustreError: 1100288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2bdb8e700 x1875681580780672/t0(0) o1000->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 LustreError: 1100288:0:(osp_object.c:615:osp_attr_get()) lustre-MDT0002-osp-MDT0000: osp_attr_get update error [0x20000000a:0x2:0x0]: rc = -5 LustreError: 1100288:0:(llog_cat.c:470:llog_cat_close()) lustre-MDT0002-osp-MDT0000: failure destroying log during cleanup: rc = -5 Lustre: lustre-MDT0000: Not available for connect from 10.240.26.185@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2c174a080 x1875681580783616/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 10.240.26.183@tcp (stopping) Lustre: Skipped 7 previous similar messages LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2c1749a00 x1875681580784512/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.26.185@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem f6f36f93-29ef-42c7-ae56-9e712347a796. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && LustreError: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: 919542:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.26.185@tcp arrived at 1788791657 with bad export cookie 1372550225497543179 LustreError: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 10.240.26.185@tcp failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.26.185@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: lustre-MDT0000: not available for connect from 10.240.26.183@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 8 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: 919543:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791661 with bad export cookie 1372550225497542745 LustreError: 919543:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: MGC10.240.26.184@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 1100814:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb283264000 x1875681580791808/t0(0) o1000->lustre-MDT0001-osp-MDT0002@10.240.26.185@tcp:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 LustreError: 1100814:0:(osp_object.c:615:osp_attr_get()) lustre-MDT0001-osp-MDT0002: osp_attr_get update error [0x20000000a:0x1:0x0]: rc = -5 LustreError: 1100814:0:(osp_object.c:615:osp_attr_get()) Skipped 1 previous similar message LustreError: 1100814:0:(llog_cat.c:470:llog_cat_close()) lustre-MDT0001-osp-MDT0002: failure destroying log during cleanup: rc = -5 LustreError: 1100814:0:(llog_cat.c:470:llog_cat_close()) Skipped 1 previous similar message LustreError: lustre-MDT0000: not available for connect from 10.240.26.185@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 10.240.26.185@tcp (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 10.240.26.185@tcp (stopping) Lustre: Skipped 8 previous similar messages LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2b1b82080 x1875681580792064/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: lustre-MDT0000: not available for connect from 10.240.26.185@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 19 previous similar messages Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? LDISKFS-fs (dm-4): unmounting filesystem 27e96d3d-0871-4d4a-a437-2ede19d3f97f. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem f6f36f93-29ef-42c7-ae56-9e712347a796 r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 16 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-80vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 27e96d3d-0871-4d4a-a437-2ede19d3f97f r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: lustre-MDT0000: Received new MDS connection from 10.240.26.185@tcp, keep former export from same NID Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null	Link to test (2026-09-07 12:04)
sanity-sec test 27a: test fileset in various nodemaps	general protection fault, probably for non-canonical address 0xf1f65d863b08da78: 0000 [#1] SMP NOPTI CPU: 0 UID: 0 PID: 408612 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: 82 c6 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 94 82 c6 0f 1f 44 00 RSP: 0018:ff5d9e4380d1fe78 EFLAGS: 00010246 RAX: ff2561021541ae40 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ff2561021541aab0 RSI: 0000000000000000 RDI: f1f65d863b08da60 RBP: ff2561021d44be00 R08: 0000000000000040 R09: 000000000000000f R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 0000096d06db666d R14: ffffffffc1ee1050 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ff2561023ba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055ae6e868164 CR3: 000000014e822006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass i2c_i801 i2c_smbus virtio_balloon pcspkr lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci virtio_net libata crct10dif_pclmul crc32_pclmul crc32c_intel net_failover ghash_clmulni_intel failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink Unloaded tainted modules: libcfs(OE):1 lnet(OE):1 ksocklnd(OE):1 obdclass(OE):1 ldiskfs(OE):1 ptlrpc(OE):1 lquota(OE):1 lraft(OE):1 mgc(OE):1 osc(OE):1 fld(OE):1 lov(OE):1 fid(OE):1 lfsck(OE):1 lod(OE):1 osd_ldiskfs(OE):1 mdt(OE):1 lmv(OE):1 mdc(OE):1 lustre(OE):1 osp(OE):1 mdd(OE):1 obdecho(OE):1 [last unloaded: libcfs(OE)]	Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset Autotest: Test running for 170 minutes (lustre-b_es-reviews_review-dne-part-2_30126.102) \| 2026-08-18 12:33:45 LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 10.240.38.53@tcp failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 10.240.38.53@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2 LustreError: MGC10.240.38.53@tcp: Connection to MGS (at 10.240.38.53@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 15 previous similar messages LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff256101c378b0c0 x1873854147156480/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.38.53@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295 LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff256101c3789a00 x1873854147157504/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.38.53@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295 LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem 30dd2e52-04d6-4abb-b4a3-b85d83f69c4c. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0001: not available for connect from 10.240.38.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 63 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 LDISKFS-fs (dm-4): unmounting filesystem 32556fd7-2935-4ace-863f-f26830ed632a. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm229.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: trevis-156vm229.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm232.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: trevis-156vm232.trevis.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing unload_modules_local LNet: 406122:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 406122:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.38.54@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm232.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm229.trevis.whamcloud.com: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: trevis-156vm232.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-156vm229.trevis.whamcloud.com: executing load_modules_local Lustre: Lustre: Build Version: 2.16.0_ddn55_1_gbb4bd4f LNet: Added LNI 10.240.38.54@tcp [8/256/0/180] Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ alg: No test for zstdfast (zstdfast-lustre-generic) alg: No test for zstdfast (zstdfast-lustre-scomp) Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem 30dd2e52-04d6-4abb-b4a3-b85d83f69c4c r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check LustreError: lustre-MDT0003: not available for connect from 10.240.38.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-08-18 10:35)
sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects	general protection fault, probably for non-canonical address 0xb9598ea836910a1b: 0000 [#1] PREEMPT SMP NOPTI CPU: 1 UID: 0 PID: 741821 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: 74 c2 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 74 74 c2 0f 1f 44 00 RSP: 0018:ff59c8a60dfdbe90 EFLAGS: 00010246 RAX: ff38b47d76a3bef0 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ff38b47d76a3b2f8 RSI: 0000000000000000 RDI: b9598ea836910a03 RBP: ff38b47d79236e00 R08: 0000000000000040 R09: 000000000000000f R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 00000f6d88ccf073 R14: ff38b47d6f527000 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ff38b47dbbb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055e4bf865fc0 CR3: 0000000139fac005 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: <TASK> ? srso_alias_return_thunk+0x5/0xfbef5 ? show_trace_log_lvl+0x255/0x2f0 ? show_trace_log_lvl+0x255/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt virtio_net iTCO_vendor_support net_failover failover pcspkr virtio_balloon i2c_i801 lpc_ich i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw [last unloaded: dm_flakey]	Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0001.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0003.raft Autotest: Test running for 280 minutes (lustre-b_es-reviews_review-dne-part-7_30092.100) \| 2026-08-17 08:32:43 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-..index_backup=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x199 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd.MDT.force_sync=1 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds2 Lustre: Failing over lustre-MDT0001 Lustre: * cfs_fail_loc=199, val=0* Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0 Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0 LDISKFS-fs (dm-3): unmounting filesystem 3ff6f387-7b80-4bdd-9a21-b667239432bd. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds4 Lustre: Failing over lustre-MDT0003 Lustre: * cfs_fail_loc=199, val=0* Lustre: Skipped 46 previous similar messages Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000001:0x3:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x3:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x4:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x5:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x6:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x7:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x8:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xd:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xe:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xf:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x10:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x11:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x12:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x13:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x14:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x15:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x16:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x17:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x18:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x19:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1a:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1b:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1c:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1d:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1f:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x20:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x21:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x22:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x23:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x24:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x25:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x26:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x27:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x28:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x29:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2a:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2b:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2c:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2d:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2e:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2f:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x1:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x2:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x3:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20001:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20002:0x0]: rc = 0 Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20003:0x0]: rc = 0 LDISKFS-fs (dm-4): unmounting filesystem cd055788-3a83-4097-bde7-f1e4c23dc312. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft -o user_xattr /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem 3ff6f387-7b80-4bdd-9a21-b667239432bd r/w with ordered data mode. Quota mode: journalled. Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote -o user_xattr /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem cd055788-3a83-4097-bde7-f1e4c23dc312 r/w with ordered data mode. Quota mode: journalled. Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2010000-MDT0003' with [0x200000005:0x3:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x10000-MDT0003' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1010000-MDT0003' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2020000-MDT0003' with [0x200000005:0x20003:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x20000-MDT0003' with [0x200000005:0x20001:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1020000-MDT0003' with [0x200000005:0x20002:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0 Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}'	Link to test (2026-08-17 04:42)
sanity-pcc test 2a: Test multi open when creating	general protection fault, probably for non-canonical address 0x8bda9b69eaffc5df: 0000 [#1] SMP PTI CPU: 0 UID: 0 PID: 99002 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffce4d478ffe78 EFLAGS: 00010246 RAX: ffff8dc184fec0a0 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8dc184fecb10 RSI: 0000000000000000 RDI: 8bda9b69eaffc5c7 RBP: ffff8dc1aaab9400 R08: 0000000000000040 R09: 0000000000000002 R10: 0000000000000000 R11: ffff8dc180f11b90 R12: 0000000000000000 R13: 00000171b79e4252 R14: ffffffffc13fc050 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ffff8dc1fbc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f490c0ea260 CR3: 0000000077022004 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfd/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel kvm mlx5_ib irqbypass rapl iTCO_wdt ib_uverbs iTCO_vendor_support pcspkr i2c_i801 macsec mlx5_fwctl fwctl i2c_smbus virtio_balloon lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core ahci libahci mlxfw libata virtio_net tls crct10dif_pclmul crc32_pclmul virtio_blk crc32c_intel ghash_clmulni_intel psample net_failover pci_hyperv_intf failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod ib_ipoib ib_cm ib_core nfnetlink	Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: lfs --list-commands Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 18 previous similar messages LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8dc1fb5a2000 x1873493963218688/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.29.73@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem fe4e1c8b-b110-4c63-a8f4-3c2fc2327a0f. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0000: not available for connect from 10.240.29.69@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 50 previous similar messages LustreError: 83390:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.29.73@tcp arrived at 1786704310 with bad export cookie 5758006088113629446 LustreError: 83390:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: MGC10.240.29.72@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8dc19f69dd40 x1873493963230464/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 6 previous similar messages Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? LDISKFS-fs (dm-4): unmounting filesystem 5d0ea3fd-b039-4036-973a-aa6b6fd8397c. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: tune2fs -O project /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: tune2fs -O project /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem fe4e1c8b-b110-4c63-a8f4-3c2fc2327a0f r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/l Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm27.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-146vm27.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 5d0ea3fd-b039-4036-973a-aa6b6fd8397c r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/l	Link to test (2026-08-14 11:42)
conf-sanity test 53a: check OSS thread count params	general protection fault, probably for non-canonical address 0xac864f24560509f4: 0000 [#1] PREEMPT SMP PTI CPU: 0 UID: 0 PID: 849745 Comm: lraft_0003 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffafb946e9fe90 EFLAGS: 00010246 RAX: ffff8a59a872ce90 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8a59a872c0d0 RSI: 0000000000000000 RDI: ac864f24560509dc RBP: ffff8a59a18eee00 R08: 0000000000000040 R09: 0000000000000004 R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 00001c4122860736 R14: ffff8a59b90b4000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8a59fbc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005643865f0ce0 CR3: 0000000103c80006 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xd2/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel ib_umad(OE) kvm sunrpc rapl virtio_net iTCO_wdt iTCO_vendor_support net_failover mlx5_ib(OE) i2c_i801 ib_uverbs(OE) failover mlx5_fwctl(OE) fwctl(OE) ib_core(OE) virtio_balloon i2c_smbus pcspkr macsec lpc_ich joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core(OE) mlxfw(OE) tls ahci crct10dif_pclmul libahci psample crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_blk mlxdevm(OE) mlx_compat(OE) pci_hyperv_intf serio_raw Unloaded tainted modules: lzstd(OE):2 llz4hc(OE):2 llz4(OE):2 libcfs(OE):3 lnet(OE):3 obdclass(OE):3 ko2iblnd(OE):3 ldiskfs(OE):3 ptlrpc(OE):3 lquota(OE):3 ptlrpc_gss(OE):2 lraft(OE):3 mgc(OE):3 osc(OE):3 fld(OE):3 lov(OE):3 fid(OE):3 lfsck(OE):3 lod(OE):3 osd_ldiskfs(OE):3 mdt(OE):3 lmv(OE):3 mdc(OE):3 lustre(OE):3 osp(OE):3 obdecho(OE):2 mdd(OE):3 [last unloaded: libcfs(OE)]	Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Autotest: Test running for 510 minutes (lustre-b_es7_0_full-dne-o2ib-part-3_315.15) Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2 LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Key type lgssc unregistered LNet: 828826:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 172.16.12.13@o2ib Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.0_ddn55 LNet: Using FastReg for registration LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180] Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ alg: No test for zstdfast (zstdfast-lustre-generic) alg: No test for zstdfast (zstdfast-lustre-scomp) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0003: Not available for connect from 172.16.12.12@o2ib (not set up) Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 172.16.12.11@o2ib failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 172.16.12.11@o2ib) was lost; in progress operations using this service will wait for recovery to complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 172.16.12.12@o2ib failed: rc = -107 Lustre: lustre-MDT0000-lwp-MDT0003: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete LustreError: Skipped 1 previous similar message Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2 LustreError: MGC172.16.12.12@o2ib: Connection to MGS (at 172.16.12.12@o2ib) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001: Not available for connect from 172.16.12.12@o2ib (stopping) Lustre: Skipped 1 previous similar message LustreError: lustre-MDT0001-osp-MDT0003: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 830049:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ff32e4cc x1873268767260160/t0(0) o1203->lustre-MDT0000-osp-MDT0001@172.16.12.12@o2ib:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 LustreError: 830050:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000601e40ee x1873268767260544/t0(0) o1203->lustre-MDT0000-osp-MDT0001@172.16.12.12@o2ib:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 LustreError: 830050:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Key type lgssc unregistered LNet: 837576:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 172.16.12.13@o2ib Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.0_ddn55 LNet: Using FastReg for registration LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180] Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0001: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 172.16.12.12@o2ib (not set up) LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 22 previous similar messages Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: lustre-OST0000-osc-MDT0003: Connection to lustre-OST0000 (at 172.16.12.11@o2ib) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 172.16.12.11@o2ib failed: rc = -107 Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete Autotest: Test running for 515 minutes (lustre-b_es7_0_full-dne-o2ib-part-3_315.15) LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 172.16.12.12@o2ib failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2 LustreError: MGC172.16.12.12@o2ib: Connection to MGS (at 172.16.12.12@o2ib) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca. Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4 LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c. Lustre: server umount lustre-MDT0003 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 845472:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 172.16.12.13@o2ib Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.0_ddn55 LNet: Using FastReg for registration LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180] Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ alg: No test for zstdfast (zstdfast-lustre-generic) alg: No test for zstdfast (zstdfast-lustre-scomp) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2 Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4 LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-08-12 01:56)
conf-sanity test 26: MDT startup failure cleans LOV (should return errs)	general protection fault, probably for non-canonical address 0x97aec02a62ccb9e7: 0000 [#1] SMP PTI CPU: 0 UID: 0 PID: 500943 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffffcefb8ca57e78 EFLAGS: 00010246 RAX: ffff8b7ac9a4e428 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8b7ac9a4e838 RSI: 0000000000000000 RDI: 97aec02a62ccb9cf RBP: ffff8b7ac1ec0400 R08: 0000000000000040 R09: 0000000000000004 R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000 R13: 000004b9e50688e1 R14: ffffffffc17e9050 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8b7b3bc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055c4d5061f50 CR3: 000000001e622003 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? __pfx_lraft_timerd+0x10/0x10 [lraft] ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x155/0x350 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xfd/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs intel_rapl_msr intel_rapl_common rfkill intel_uncore_frequency_common mlx5_ib kvm_intel ib_uverbs kvm macsec mlx5_fwctl irqbypass fwctl rapl iTCO_wdt i2c_i801 iTCO_vendor_support lpc_ich pcspkr virtio_balloon i2c_smbus joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core virtio_net ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata mlxfw ghash_clmulni_intel net_failover failover tls virtio_blk psample pci_hyperv_intf serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod ib_ipoib ib_cm ib_core nfnetlink [last unloaded: dm_flakey]	Lustre: DEBUG MARKER: lsmod \| grep -q lustre \|\| modprobe lustre Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000135 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem 296314a3-b64c-41ae-a550-f4948704dd1a r/w with ordered data mode. Quota mode: journalled. Lustre: * cfs_fail_loc=135, val=0* LustreError: 497090:0:(obd_config.c:801:class_setup()) setup lustre-MDT0000 failed (-2) LustreError: 497090:0:(obd_config.c:2032:class_config_llog_handler()) MGC10.240.26.60@tcp: cfg command failed: rc = -2 Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f LustreError: MGC10.240.26.60@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info LustreError: 497051:0:(tgt_mount.c:1843:server_start_targets()) failed to start server lustre-MDT0000: -2 LustreError: 497051:0:(tgt_mount.c:2587:server_fill_super()) Unable to start targets: -2 LustreError: 497051:0:(obd_config.c:854:class_cleanup()) Device 5 not setup Lustre: server umount lustre-MDT0000 complete LustreError: 497051:0:(super25.c:206:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2 LDISKFS-fs (dm-4): unmounting filesystem 296314a3-b64c-41ae-a550-f4948704dd1a. Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem 296314a3-b64c-41ae-a550-f4948704dd1a r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm33.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-141vm33.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-5): mounted filesystem 1f49507d-b08c-47f6-8b38-f16a412a333a r/w with ordered data mode. Quota mode: journalled. Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-08-10 09:51)
sanity-sec test 27ac: test nodemap llog and IAM fileset compatibility	general protection fault, probably for non-canonical address 0x8859eafdb3f5b2d6: 0000 [#1] PREEMPT SMP PTI CPU: 1 UID: 0 PID: 596278 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1 Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:raft_node_is_active+0x9/0x20 [lraft] Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00 RSP: 0018:ffff99c3c5bf3e90 EFLAGS: 00010246 RAX: ffff8a158ded4b80 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8a158ded4cd8 RSI: 0000000000000000 RDI: 8859eafdb3f5b2be RBP: ffff8a158544ac00 R08: 0000000000000040 R09: 0000000000000002 R10: 0000000000000000 R11: ffff8a15fbc4d0d0 R12: 0000000000000000 R13: 000007f0e1b082c5 R14: ffff8a15b4db4000 R15: 0000000000000007 FS: 0000000000000000(0000) GS:ffff8a15fbd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000556ac7ccef70 CR3: 0000000102716003 CR4: 0000000000372ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? show_trace_log_lvl+0x1b0/0x2f0 ? show_trace_log_lvl+0x1b0/0x2f0 ? raft_get_num_voting_nodes+0x34/0x70 [lraft] ? __die_body.cold+0x8/0x12 ? die_addr+0x3c/0x60 ? exc_general_protection+0x178/0x410 ? asm_exc_general_protection+0x26/0x30 ? raft_node_is_active+0x9/0x20 [lraft] raft_get_num_voting_nodes+0x34/0x70 [lraft] lraft_timerd+0x153/0x320 [lraft] ? __pfx_lraft_timerd+0x10/0x10 [lraft] kthread+0xd2/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x34/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill mlx5_ib ib_uverbs macsec intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel mlx5_core sunrpc kvm mlxfw psample iTCO_wdt rapl iTCO_vendor_support tls virtio_net net_failover i2c_i801 pci_hyperv_intf failover lpc_ich pcspkr virtio_balloon i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core Unloaded tainted modules: lzstd(OE):3 llz4hc(OE):3 llz4(OE):3 ptlrpc_gss(OE):3 obdecho(OE):3 libcfs(OE):4 lnet(OE):4 ksocklnd(OE):4 obdclass(OE):4 ldiskfs(OE):4 ptlrpc(OE):4 lquota(OE):4 lraft(OE):4 mgc(OE):4 mgs(OE):4 osc(OE):4 fld(OE):4 lov(OE):4 fid(OE):4 lfsck(OE):4 lod(OE):4 osd_ldiskfs(OE):4 mdt(OE):4 lmv(OE):4 mdc(OE):4 lustre(OE):4 osp(OE):4 mdd(OE):4 [last unloaded: lzstd(OE)]	Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add test27ac_nm Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset /iam_fileset Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del test27ac_nm ; /usr/sbin/lctl nodemap_add test27ac_nm Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl set_param nodemap.test27ac_nm.fileset=/llog_fileset Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.test27ac_nm.fileset=/llog_fileset Lustre: Setting parameter general.nodemap.test27ac_nm.fileset=/llog_fileset in log params Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping) Lustre: Skipped 18 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15aa835d40 x1872805408788864/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages LDISKFS-fs (dm-3): unmounting filesystem b785b3f1-096b-49a6-83f7-964da059cf10. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: 555405:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.70@tcp arrived at 1786047055 with bad export cookie 13200158399717358067 LustreError: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 10.240.27.70@tcp failed: rc = -107 LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: 555405:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047063 with bad export cookie 13200158399717357633 LustreError: MGC10.240.27.69@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15aaa0d6c0 x1872805408799232/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 4 previous similar messages LDISKFS-fs (dm-4): unmounting filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Key type lgssc unregistered LNet: 572521:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 572521:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.27.69@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.0_ddn55_2_g66cf7cf LNet: Added LNI 10.240.27.69@tcp [8/256/0/180] Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ alg: No test for zstdfast (zstdfast-lustre-generic) alg: No test for zstdfast (zstdfast-lustre-scomp) Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem b785b3f1-096b-49a6-83f7-964da059cf10 r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3 Lustre: 581109:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: 581225:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Autotest: Test running for 135 minutes (lustre-b_es-reviews_review-dne-exa7-part-1_29794.157) Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset \| awk '/primary/ { if ($3 == "/llog_fileset") print $3 }' Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset /iam_fileset Lustre: Disabling parameter general.nodemap.test27ac_nm.fileset= in log params Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl set_param nodemap.test27ac_nm.fileset=/llog_fileset Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset clear Lustre: 585940:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/nodemap.test27ac_nm.fileset= (mode = 1) failed: rc = -2 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids \| grep -w tcp \| cut -f 1 -d @ Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: lustre-MDT0000: Not available for connect from 10.240.27.70@tcp (stopping) Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a158d6eb400 x1872806155674368/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping) Lustre: Skipped 6 previous similar messages LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15fb58ecc0 x1872806155675392/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.27.70@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 10.240.27.70@tcp (stopping) Lustre: Skipped 6 previous similar messages LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15ab154340 x1872806155678848/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping) Lustre: Skipped 5 previous similar messages LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15ab156d80 x1872806155680640/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295 LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping) Lustre: Skipped 13 previous similar messages Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? LDISKFS-fs (dm-3): unmounting filesystem b785b3f1-096b-49a6-83f7-964da059cf10. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 576098:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.70@tcp arrived at 1786047274 with bad export cookie 774761076203025042 LustreError: lustre-MDT0001-osp-MDT0002: operation raft_appendentries to node 10.240.27.70@tcp failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.27.70@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 LustreError: 576099:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047282 with bad export cookie 774761076203024601 LustreError: MGC10.240.27.69@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15fb4e6000 x1872806155692416/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295 LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 4 previous similar messages LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 24 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 10.240.27.70@tcp (stopping) Lustre: Skipped 7 previous similar messages LDISKFS-fs (dm-4): unmounting filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6. Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local Key type lgssc unregistered LNet: 589984:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 589984:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.27.69@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.0_ddn55_2_g66cf7cf LNet: Added LNI 10.240.27.69@tcp [8/256/0/180] Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ alg: No test for zstdfast (zstdfast-lustre-generic) alg: No test for zstdfast (zstdfast-lustre-scomp) Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-3): mounted filesystem b785b3f1-096b-49a6-83f7-964da059cf10 r/w with ordered data mode. Quota mode: journalled. LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null LustreError: lustre-MDT0002: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6 r/w with ordered data mode. Quota mode: journalled. Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u	Link to test (2026-08-06 18:27)

Failing Test

Full Crash

Messages before crash

Link and date

sanity-scrub test 10a: non-stopped OI scrub should auto restarts after MDS remount (1)

general protection fault, probably for non-canonical address 0xca2c8ecb4093b597: 0000 [#1] PREEMPT SMP PTI
CPU: 1 UID: 0 PID: 1143263 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffae4d062cfe90 EFLAGS: 00010202
RAX: ffff9b02a3f2ad90 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff9b029cf1c140 RSI: 0000000000000000 RDI: ca2c8ecb4093b57f
RBP: ffff9b02e84d3c00 R08: 0000000000000040 R09: 0000000000000002
R10: 0000000000000000 R11: ffff9b0288f28090 R12: 0000000000000001
R13: 0000128489363d72 R14: ffff9b02b4b89000 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff9b02fbd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffc1b8ebe18 CR3: 000000012b0ea004 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xd2/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs mlx5_ib ib_uverbs macsec rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel mlx5_core kvm sunrpc rapl iTCO_wdt iTCO_vendor_support mlxfw psample tls pcspkr virtio_net i2c_i801 lpc_ich i2c_smbus pci_hyperv_intf virtio_balloon net_failover failover joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core [last unloaded: dm_flakey]
Crash core archive creation started: kernel-crash.onyx-146vm8.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-146vm8.vmcore.tar.gz

Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-143vm34.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-143vm34.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: lctl get_param -n timeout
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush
Lustre: 1127397:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
Lustre: 1127397:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 5 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Autotest: Test running for 335 minutes (lustre-b_es-reviews_review-dne-part-7_31815.121) | 2026-10-23 13:33:47
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 84da455e-b011-40e6-a474-a82effd5ddd2.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 84da455e-b011-40e6-a474-a82effd5ddd2 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 84da455e-b011-40e6-a474-a82effd5ddd2.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=400000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem a46b8bbe-a1cb-418f-8c06-d798713cbe7e.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds3_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.54@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=400000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds3_flakey
LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey lustre-MDT0002
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-*.*.auto_scrub=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-*.*.full_scrub_ratio=0
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x190
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200004284:0x1:0x0]/40: rc = 0
Lustre: *** cfs_fail_loc=190, val=1***
Lustre: Skipped 56 previous similar messages
Lustre: lustre-MDT0002: trigger OI scrub by RPC for [0x280004281:0x1:0x0]/250 with flags 0x52: rc = 0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub |
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 40c13cdc-5cf0-4575-8b4c-b78e23c37488 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem d10909a1-d7ae-4e37-adeb-a1409150ced4 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm8.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm28.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub |

Link to test (2026-10-23 09:46)

conf-sanity test 50c: lazystatfs one server down

general protection fault, probably for non-canonical address 0xfe1c15db5069ac8e: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 UID: 0 PID: 587560 Comm: lraft_0003 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: f0 d1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 f4 f0 d1 0f 1f 44 00
RSP: 0018:ff63da388aa2be90 EFLAGS: 00010246
RAX: ff407914c4d96758 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ff407914c4d96110 RSI: 0000000000000000 RDI: fe1c15db5069ac76
RBP: ff407914c2b36200 R08: 0000000000000040 R09: 0000000000000004
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 00000d156ad7fc04 R14: ff407914cf0af000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ff4079153ba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000564965aad164 CR3: 00000001315d2002 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xcf/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: lzstd(OE) llz4hc(OE) llz4(OE) obdecho(OE) ec(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support virtio_net net_failover i2c_i801 lpc_ich virtio_balloon failover pcspkr i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_blk serio_raw
Crash core archive creation started: kernel-crash.trevis-152vm163.vmcore.tar.gz
Crash core archive created: kernel-crash.trevis-152vm163.vmcore.tar.gz

Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem dd31694d-9434-473e-bda8-dde26d69ae88 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: trevis-152vm163.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: trevis-152vm156.trevis.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 590f1cbc-f830-4898-af83-0b17f3dd0c00 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-10-13 12:59)

sanity-sec test 27ab: test nodemap idmap offset

general protection fault, probably for non-canonical address 0x82c931d33a241cd6: 0000 [#1] PREEMPT SMP PTI
CPU: 0 UID: 0 PID: 549916 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffac4c4b563e90 EFLAGS: 00010246
RAX: ffff9ba7c1daff18 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff9ba7c1daf968 RSI: 0000000000000000 RDI: 82c931d33a241cbe
RBP: ffff9ba7ea2bc400 R08: 0000000000000040 R09: 000000000000000f
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 00000818885cff29 R14: ffff9ba7e3654000 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ffff9ba83bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fff07209e88 CR3: 0000000123464001 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xd2/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs mlx5_ib ib_uverbs macsec rfkill intel_rapl_msr intel_rapl_common mlx5_core intel_uncore_frequency_common kvm_intel sunrpc kvm iTCO_wdt iTCO_vendor_support rapl virtio_net pcspkr net_failover mlxfw psample failover virtio_balloon tls i2c_i801 lpc_ich i2c_smbus pci_hyperv_intf joydev loop fuse dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core
Unloaded tainted modules: lzstd(OE):1 llz4hc(OE):1 llz4(OE):1 obdecho(OE):1 libcfs(OE):2 lnet(OE):2 ksocklnd(OE):2 obdclass(OE):2 ldiskfs(OE):2 ptlrpc(OE):2 lquota(OE):2 ptlrpc_gss(OE):2 lraft(OE):2 mgc(OE):2 mgs(OE):2 osc(OE):2 fld(OE):2 lov(OE):2 fid(OE):2 lfsck(OE):2 lod(OE):2 osd_ldiskfs(OE):2 mdt(OE):2 lmv(OE):2 mdc(OE):2 lustre(OE):2 osp(OE):2 mdd(OE):2 [last unloaded: lzstd(OE)]
Crash core archive creation started: kernel-crash.onyx-141vm38.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-141vm38.vmcore.tar.gz

Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add Test18109
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add OffsetTest
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_offset --name Test18109 --offset 100000 --limit 200000
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_offset --name OffsetTest --offset 150000 --limit 100000
LustreError: 530213:0:(nodemap_handler.c:5256:nodemap_add_offset()) OffsetTest: new offset 150000+100000 overlaps with existing nodemap Test18109 offset 100000+200000: rc = -34
LustreError: 530213:0:(nodemap_handler.c:6589:server_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE04C for OffsetTest: rc = -34
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name Test18109 --idtype uid --idmap 500-509:0-9
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.idmap
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name Test18109 --range 1.1.1.1@tcp777
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property squash_uid --value 65534
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property squash_gid --value 65534
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_idmap --name Test18109 --idtype uid --idmap 500-509:0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name Test18109 --idtype uid --idmap 500-509:200000-200010
LustreError: 533554:0:(nodemap_handler.c:753:nodemap_parse_idmap()) Test18109: range length mismatch between client id 500-509 and fs id 200000-200010: rc = -22
LustreError: 533554:0:(nodemap_handler.c:6589:server_iocontrol_nodemap()) MGS: OBD_IOC_NODEMAP command CE044 for Test18109: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.idmap
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property trusted --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 501
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name Test18109 --property admin --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype uid --id 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_test_id --nid 1.1.1.1@tcp777 --idtype gid --id 0
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
Lustre: lustre-MDT0000: Not available for connect from 10.240.28.195@tcp (stopping)
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
Lustre: Skipped 2 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
LustreError: lustre-MDT0000: not available for connect from 10.240.26.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: 514712:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.28.195@tcp arrived at 1791550016 with bad export cookie 1158799906114673421
Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.28.195@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 21 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: 514713:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1791550025 with bad export cookie 1158799906114672980
LustreError: 514713:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages
LustreError: MGC10.240.26.66@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 514722:0:(mgc_request.c:616:do_requeue()) failed processing log: -5
Lustre: lustre-MDT0002: Not available for connect from 10.240.26.35@tcp (stopping)
Lustre: Skipped 2 previous similar messages
LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7db169c00 x1878576097322112/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LDISKFS-fs (dm-4): unmounting filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 11 previous similar messages
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: 542209:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
LustreError: lustre-OST0007-osc-MDT0000: operation ost_connect to node 10.240.26.35@tcp failed: rc = -11
LustreError: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-180vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: lctl get_param -n timeout
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush
Lustre: 547102:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
Lustre: 547102:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del_offset --name Test18109
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.Test18109.offset
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7f1d6fb80 x1878576097657728/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.28.195@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
Lustre: Skipped 6 previous similar messages
LustreError: 513733:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7db176d80 x1878576097658624/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.28.195@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 513733:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LDISKFS-fs (dm-3): unmounting filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0000: not available for connect from 10.240.28.195@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 7 previous similar messages
LustreError: 537203:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.28.195@tcp arrived at 1791550248 with bad export cookie 1158799906114676410
LustreError: 537203:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages
Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.28.195@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: 542245:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1791550257 with bad export cookie 1158799906114675990
LustreError: 542245:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 4 previous similar messages
LustreError: MGC10.240.26.66@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ba7da823100 x1878576097670528/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LustreError: 513734:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: lustre-MDT0002: Not available for connect from 10.240.28.195@tcp (stopping)
Lustre: Skipped 14 previous similar messages
LDISKFS-fs (dm-4): unmounting filesystem ddd884c7-1273-414c-a6a1-c3b20fc0bdd2.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Autotest: Test running for 145 minutes (lustre-b_es-reviews_review-dne-exa7-part-1_31463.223) | 2026-10-09 12:51:13
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 96bc8b6f-3e4b-462a-b0f5-a0089ac9cf5d r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 33 previous similar messages
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-141vm38.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-106vm12.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all

Link to test (2026-10-09 11:43)

sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases

general protection fault, probably for non-canonical address 0x9cb60657e6389348: 0000 [#1] SMP NOPTI
CPU: 1 UID: 0 PID: 1048661 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: e7 f8 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 74 e7 f8 0f 1f 44 00
RSP: 0018:ff664d7a4d55fe78 EFLAGS: 00010202
RAX: ff3515cca7a59bc0 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ff3515cca624b2e8 RSI: 0000000000000000 RDI: 9cb60657e6389330
RBP: ff3515ccc03d1400 R08: 0000000000000040 R09: 000000000000000f
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000001
R13: 00000f9238e69bb3 R14: ffffffffc1e93050 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ff3515ccfbb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffcc7fa7578 CR3: 0000000137f5c006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? srso_alias_return_thunk+0x5/0xfbef5
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
? srso_alias_return_thunk+0x5/0xfbef5
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass pcspkr i2c_i801 lpc_ich i2c_smbus virtio_balloon joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata virtio_net crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel net_failover failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink [last unloaded: dm_flakey]
Crash core archive creation started: kernel-crash.onyx-156vm44.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-156vm44.vmcore.tar.gz

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: invalid oi count 63, remove them, then set it to 64
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: lustre-MDT0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted.
Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4
Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub
Autotest: Test running for 280 minutes (lustre-b_es-reviews_review-dne-part-7_31396.128) | 2026-10-05 21:15:11
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: 1028745:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17
Lustre: 1028745:0:(mgs_llog.c:1439:mgs_modify_param()) Skipped 2 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: lctl get_param -n timeout
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.28.95@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
Lustre: Failing over lustre-MDT0002
LustreError: Skipped 1 previous similar message
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.2
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.4
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.8
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.16
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.32
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64
Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: invalid oi count 58, remove them, then set it to 64
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: lustre-MDT0000: Recovery over after 0:17, of 3 clients 3 recovered and 0 were evicted.
Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 4
Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 4
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm42.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: lctl get_param -n timeout
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
LustreError: lustre-MDT0000: not available for connect from 10.240.28.94@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 73 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 8 previous similar messages
LustreError: 1034605:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.244@tcp arrived at 1791235114 with bad export cookie 10353082178918664466
LustreError: 1034605:0:(ldlm_lockd.c:2572:ldlm_cancel_handler()) Skipped 9 previous similar messages
LustreError: lustre-MDT0001-osp-MDT0002: operation raft_appendentries to node 10.240.27.244@tcp failed: rc = -107
LustreError: Skipped 10 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.3
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.9
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.27
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 57dec75a-d8c3-443f-9a26-8e61296f7196 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 5 previous similar messages
Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 0fe5667f-f4b3-4c74-9e3e-9b554ffe92a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: invalid oi count 56, remove them, then set it to 64
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm44.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm255.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
Lustre: lustre-MDT0000: Recovery over after 0:18, of 3 clients 3 recovered and 0 were evicted.
Lustre: lustre-MDT0000-osp-MDT0002: Connection restored to 10.240.28.95@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub

Link to test (2026-10-05 17:31)

conf-sanity test 113: Shadow mountpoint correctly report ro/rw for mounts

general protection fault, probably for non-canonical address 0xa0f272be9ded70a0: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 UID: 0 PID: 1532905 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: 31 d4 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 04 31 d4 0f 1f 44 00
RSP: 0018:ff6afccc0e933e90 EFLAGS: 00010202
RAX: ff2ab58904f4c7c0 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ff2ab5892c5e26f8 RSI: 0000000000000000 RDI: a0f272be9ded7088
RBP: ff2ab58936017400 R08: 0000000000000040 R09: 0000000000000004
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000001
R13: 00001b7bc7467ea5 R14: ff2ab589030d5008 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ff2ab5897bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f2437d5048 CR3: 000000011fb3e002 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xcf/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support pcspkr virtio_net lpc_ich i2c_i801 virtio_balloon net_failover failover i2c_smbus joydev loop fuse dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw
Unloaded tainted modules: lzstd(OE):12 llz4hc(OE):12 llz4(OE):12 libcfs(OE):21 lnet(OE):14 ksocklnd(OE):14 obdclass(OE):14 ldiskfs(OE):14 ptlrpc(OE):14 lquota(OE):14 ptlrpc_gss(OE):14 lraft(OE):14 mgc(OE):14 osc(OE):14 fld(OE):14 lov(OE):14 fid(OE):14 lfsck(OE):14 lod(OE):14 osd_ldiskfs(OE):14 mdt(OE):14 lmv(OE):14 mdc(OE):14 lustre(OE):14 osp(OE):14 obdecho(OE):12 mdd(OE):14 [last unloaded: lzstd(OE)]
Crash core archive creation started: kernel-crash.onyx-157vm25.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-157vm25.vmcore.tar.gz

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 21718c17-d480-4cef-9115-6dd7dcd008ec r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-157vm25.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-157vm24.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;

Link to test (2026-09-26 17:04)

replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1

general protection fault, probably for non-canonical address 0x2c113f106f93cfea: 0000 [#1] SMP NOPTI
CPU: 1 UID: 0 PID: 213438 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: 8a dd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 c4 8a dd 0f 1f 44 00
RSP: 0018:ff5293678beefe78 EFLAGS: 00010202
RAX: ff396b5c6cde1b50 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ff396b5c6cc25e38 RSI: 0000000000000000 RDI: 2c113f106f93cfd2
RBP: ff396b5c6d63be00 R08: 0000000000000040 R09: 0000000000000002
R10: 0000000000000000 R11: ff396b5c43980090 R12: 0000000000000001
R13: 000006d9cab2c23a R14: ffffffffc1c5e050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ff396b5cbbb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c13b6e1158 CR3: 0000000132aaa006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
? srso_alias_return_thunk+0x5/0xfbef5
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass pcspkr i2c_i801 virtio_balloon lpc_ich i2c_smbus joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 virtio_net crct10dif_pclmul ahci crc32_pclmul libahci crc32c_intel ghash_clmulni_intel libata net_failover virtio_blk failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink
Crash core archive creation started: kernel-crash.onyx-156vm171.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-156vm171.vmcore.tar.gz

Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701
Lustre: *** cfs_fail_loc=1701, val=2147483648***
LustreError: 28971:0:(ldlm_lib.c:3394:target_send_reply_msg()) @@@ dropping reply req@ff396b5c67d5b100 x1877403953518848/t64424509950(0) o1000->lustre-MDT0000-mdtlov_UUID@10.240.24.18@tcp:497/0 lens 1320/4320 e 0 to 0 dl 1790439207 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up1-0.0' uid:0 gid:0 projid:4294967295
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds2
Lustre: Failing over lustre-MDT0001
LDISKFS-fs (dm-3): unmounting filesystem b7c698fa-8b94-4eaa-bacb-dbd6c9d92458.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 145 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem b7c698fa-8b94-4eaa-bacb-dbd6c9d92458 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 2 previous similar messages
Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
Lustre: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 5 clients reconnect
Lustre: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-09-26 13:23)

sanity-sec test 62: e2fsck with encrypted files

general protection fault, probably for non-canonical address 0x4e895d66dee964d2: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 561185 Comm: lraft_0003 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: 74 e2 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 44 74 e2 0f 1f 44 00
RSP: 0018:ff8b8b95cadefe78 EFLAGS: 00010246
RAX: ff48a559100f7ef0 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ff48a559100f7658 RSI: 0000000000000000 RDI: 4e895d66dee964ba
RBP: ff48a5590096e000 R08: 0000000000000040 R09: 0000000000000002
R10: 0000000000000000 R11: ff48a5590613c390 R12: 0000000000000000
R13: 00000d8d21c89d76 R14: ffffffffc19c6050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ff48a5593ba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffdff591a48 CR3: 000000010b1da006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass i2c_i801 pcspkr i2c_smbus virtio_balloon lpc_ich joydev loop fuse vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata virtio_net crct10dif_pclmul crc32_pclmul net_failover crc32c_intel failover ghash_clmulni_intel virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink
Unloaded tainted modules: lzstd(OE):6 llz4hc(OE):6 llz4(OE):6 libcfs(OE):7 lnet(OE):7 ksocklnd(OE):7 obdclass(OE):7 ldiskfs(OE):7 ptlrpc(OE):7 lquota(OE):7 ptlrpc_gss(OE):7 lraft(OE):7 mgc(OE):7 osc(OE):7 fld(OE):7 lov(OE):7 fid(OE):7 lfsck(OE):7 lod(OE):7 osd_ldiskfs(OE):7 mdt(OE):7 lmv(OE):7 mdc(OE):7 lustre(OE):7 osp(OE):7 mdd(OE):7 obdecho(OE):7 [last unloaded: lzstd(OE)]
Crash core archive creation started: kernel-crash.onyx-156vm121.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-156vm121.vmcore.tar.gz

LustreError: lustre-MDT0000-osp-MDT0003: operation mds_statfs to node 10.240.30.18@tcp failed: rc = -107
Lustre: lustre-MDT0000-osp-MDT0003: Connection to lustre-MDT0000 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 10.240.30.18@tcp failed: rc = -107
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2
LustreError: MGC10.240.30.18@tcp: Connection to MGS (at 10.240.30.18@tcp) was lost; in progress operations using this service will fail
Lustre: lustre-MDT0001: Not available for connect from 10.240.30.18@tcp (stopping)
LustreError: 525337:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff48a558fa5c5f80 x1877192418673920/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.30.18@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295
LustreError: 525337:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: lustre-MDT0001-osp-MDT0003: operation mds_statfs to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
LustreError: 525338:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff48a558fa5c5c00 x1877192418675456/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.30.18@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295
LustreError: 525338:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
Lustre: Skipped 8 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 042f26d0-cd46-4437-9b82-841a32be1870.
LustreError: lustre-MDT0001: not available for connect from 10.240.30.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0001: not available for connect from 10.240.30.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 7 previous similar messages
LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 10.240.30.18@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
LustreError: lustre-MDT0001: not available for connect from 10.240.30.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0003: Not available for connect from 10.240.30.17@tcp (stopping)
LDISKFS-fs (dm-4): unmounting filesystem 40b86320-81f7-442c-aae2-8bcdc7f62b06.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 042f26d0-cd46-4437-9b82-841a32be1870 r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 7 previous similar messages
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm121.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
LustreError: lustre-MDT0003: not available for connect from 10.240.30.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm120.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 40b86320-81f7-442c-aae2-8bcdc7f62b06 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-09-24 04:33)

sanity-sec test 75b: test resource ID repair

general protection fault, probably for non-canonical address 0xeb70f32fdccb6da2: 0000 [#1] SMP PTI
CPU: 0 UID: 0 PID: 1102026 Comm: lraft_0000 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffd27f4a803e78 EFLAGS: 00010246
RAX: ffff8cb2aa1e7c38 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8cb2aa1e7cb8 RSI: 0000000000000000 RDI: eb70f32fdccb6d8a
RBP: ffff8cb2a8221600 R08: 0000000000000040 R09: 000000000000000f
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 00000b087f62bc9a R14: ffffffffc17d6050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ffff8cb2fbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5b5cecec40 CR3: 0000000138694003 CR4: 0000000000172ef0
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ec(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_intel kvm irqbypass rapl iTCO_wdt iTCO_vendor_support pcspkr virtio_balloon i2c_i801 i2c_smbus lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 virtio_net ahci libahci net_failover libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink
Unloaded tainted modules: lzstd(OE):6 llz4hc(OE):6 llz4(OE):6 ptlrpc_gss(OE):6 obdecho(OE):6 libcfs(OE):7 lnet(OE):7 ksocklnd(OE):7 obdclass(OE):7 ldiskfs(OE):7 ptlrpc(OE):7 lquota(OE):7 lraft(OE):7 mgc(OE):7 mgs(OE):7 osc(OE):7 fld(OE):7 lov(OE):7 fid(OE):7 lfsck(OE):7 lod(OE):7 osd_ldiskfs(OE):7 mdt(OE):7 lmv(OE):7 mdc(OE):7 lustre(OE):7 osp(OE):7 mdd(OE):7 [last unloaded: lzstd(OE)]
Crash core archive creation started: kernel-crash.onyx-80vm9.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-80vm9.vmcore.tar.gz

Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed
Lustre: DEBUG MARKER: sync; sync; echo 3 > /proc/sys/vm/drop_caches
bash (1099989): drop_caches: 3
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
LustreError: 1100288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2bdb8e700 x1875681580780672/t0(0) o1000->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295
LustreError: 1100288:0:(osp_object.c:615:osp_attr_get()) lustre-MDT0002-osp-MDT0000: osp_attr_get update error [0x20000000a:0x2:0x0]: rc = -5
LustreError: 1100288:0:(llog_cat.c:470:llog_cat_close()) lustre-MDT0002-osp-MDT0000: failure destroying log during cleanup: rc = -5
Lustre: lustre-MDT0000: Not available for connect from 10.240.26.185@tcp (stopping)
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
Lustre: Skipped 2 previous similar messages
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2c174a080 x1875681580783616/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Not available for connect from 10.240.26.183@tcp (stopping)
Lustre: Skipped 7 previous similar messages
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2c1749a00 x1875681580784512/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.26.185@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem f6f36f93-29ef-42c7-ae56-9e712347a796.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
LustreError: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: 919542:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.26.185@tcp arrived at 1788791657 with bad export cookie 1372550225497543179
LustreError: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 10.240.26.185@tcp failed: rc = -107
Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.26.185@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-MDT0000: not available for connect from 10.240.26.183@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 8 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: 919543:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791661 with bad export cookie 1372550225497542745
LustreError: 919543:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) Skipped 4 previous similar messages
LustreError: MGC10.240.26.184@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 1100814:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb283264000 x1875681580791808/t0(0) o1000->lustre-MDT0001-osp-MDT0002@10.240.26.185@tcp:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295
LustreError: 1100814:0:(osp_object.c:615:osp_attr_get()) lustre-MDT0001-osp-MDT0002: osp_attr_get update error [0x20000000a:0x1:0x0]: rc = -5
LustreError: 1100814:0:(osp_object.c:615:osp_attr_get()) Skipped 1 previous similar message
LustreError: 1100814:0:(llog_cat.c:470:llog_cat_close()) lustre-MDT0001-osp-MDT0002: failure destroying log during cleanup: rc = -5
LustreError: 1100814:0:(llog_cat.c:470:llog_cat_close()) Skipped 1 previous similar message
LustreError: lustre-MDT0000: not available for connect from 10.240.26.185@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 5 previous similar messages
Lustre: lustre-MDT0002: Not available for connect from 10.240.26.185@tcp (stopping)
Lustre: Skipped 4 previous similar messages
Lustre: lustre-MDT0002: Not available for connect from 10.240.26.185@tcp (stopping)
Lustre: Skipped 8 previous similar messages
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cb2b1b82080 x1875681580792064/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LustreError: 851288:0:(client.c:1303:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: lustre-MDT0000: not available for connect from 10.240.26.185@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 19 previous similar messages
Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
LDISKFS-fs (dm-4): unmounting filesystem 27e96d3d-0871-4d4a-a437-2ede19d3f97f.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem f6f36f93-29ef-42c7-ae56-9e712347a796 r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 16 previous similar messages
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 27e96d3d-0871-4d4a-a437-2ede19d3f97f r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-80vm9.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: lustre-MDT0000: Received new MDS connection from 10.240.26.185@tcp, keep former export from same NID
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null

Link to test (2026-09-07 12:04)

sanity-sec test 27a: test fileset in various nodemaps

general protection fault, probably for non-canonical address 0xf1f65d863b08da78: 0000 [#1] SMP NOPTI
CPU: 0 UID: 0 PID: 408612 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: 82 c6 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 94 82 c6 0f 1f 44 00
RSP: 0018:ff5d9e4380d1fe78 EFLAGS: 00010246
RAX: ff2561021541ae40 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ff2561021541aab0 RSI: 0000000000000000 RDI: f1f65d863b08da60
RBP: ff2561021d44be00 R08: 0000000000000040 R09: 000000000000000f
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 0000096d06db666d R14: ffffffffc1ee1050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ff2561023ba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ae6e868164 CR3: 000000014e822006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support irqbypass i2c_i801 i2c_smbus virtio_balloon pcspkr lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci virtio_net libata crct10dif_pclmul crc32_pclmul crc32c_intel net_failover ghash_clmulni_intel failover virtio_blk serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod nfnetlink
Unloaded tainted modules: libcfs(OE):1 lnet(OE):1 ksocklnd(OE):1 obdclass(OE):1 ldiskfs(OE):1 ptlrpc(OE):1 lquota(OE):1 lraft(OE):1 mgc(OE):1 osc(OE):1 fld(OE):1 lov(OE):1 fid(OE):1 lfsck(OE):1 lod(OE):1 osd_ldiskfs(OE):1 mdt(OE):1 lmv(OE):1 mdc(OE):1 lustre(OE):1 osp(OE):1 mdd(OE):1 obdecho(OE):1 [last unloaded: libcfs(OE)]

Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset
Autotest: Test running for 170 minutes (lustre-b_es-reviews_review-dne-part-2_30126.102) | 2026-08-18 12:33:45
LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 10.240.38.53@tcp failed: rc = -107
Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 10.240.38.53@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 5 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2
LustreError: MGC10.240.38.53@tcp: Connection to MGS (at 10.240.38.53@tcp) was lost; in progress operations using this service will fail
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
Lustre: Skipped 15 previous similar messages
LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff256101c378b0c0 x1873854147156480/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.38.53@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295
LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff256101c3789a00 x1873854147157504/t0(0) o1203->lustre-MDT0000-osp-MDT0001@10.240.38.53@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0001.0' uid:0 gid:0 projid:4294967295
LustreError: 7792:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 30dd2e52-04d6-4abb-b4a3-b85d83f69c4c.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0001: not available for connect from 10.240.38.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 63 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
LDISKFS-fs (dm-4): unmounting filesystem 32556fd7-2935-4ace-863f-f26830ed632a.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm229.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-156vm229.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm232.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-156vm232.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing unload_modules_local
LNet: 406122:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNetError: 406122:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found
LNet: Removed LNI 10.240.38.54@tcp
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm234.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm232.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-156vm234.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm229.trevis.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: trevis-156vm232.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-156vm229.trevis.whamcloud.com: executing load_modules_local
Lustre: Lustre: Build Version: 2.16.0_ddn55_1_gbb4bd4f
LNet: Added LNI 10.240.38.54@tcp [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm233.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm233.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 30dd2e52-04d6-4abb-b4a3-b85d83f69c4c r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
LustreError: lustre-MDT0003: not available for connect from 10.240.38.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-08-18 10:35)

sanity-scrub test 16: Initial OI scrub can rebuild crashed index objects

general protection fault, probably for non-canonical address 0xb9598ea836910a1b: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 UID: 0 PID: 741821 Comm: lraft_0001 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: 74 c2 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 e9 50 74 74 c2 0f 1f 44 00
RSP: 0018:ff59c8a60dfdbe90 EFLAGS: 00010246
RAX: ff38b47d76a3bef0 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ff38b47d76a3b2f8 RSI: 0000000000000000 RDI: b9598ea836910a03
RBP: ff38b47d79236e00 R08: 0000000000000040 R09: 000000000000000f
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 00000f6d88ccf073 R14: ff38b47d6f527000 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ff38b47dbbb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e4bf865fc0 CR3: 0000000139fac005 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x255/0x2f0
? show_trace_log_lvl+0x255/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xcf/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt virtio_net iTCO_vendor_support net_failover failover pcspkr virtio_balloon i2c_i801 lpc_ich i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw [last unloaded: dm_flakey]

Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0001.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0003.raft
Autotest: Test running for 280 minutes (lustre-b_es-reviews_review-dne-part-7_30092.100) | 2026-08-17 08:32:43
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm6.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd-*.*.index_backup=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x199
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds2
Lustre: Failing over lustre-MDT0001
Lustre: *** cfs_fail_loc=199, val=0***
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000001:0x3:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x3:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x4:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x5:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x6:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x7:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x8:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xd:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xe:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0xf:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x10:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x11:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x12:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x13:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x14:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x15:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x16:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x17:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x18:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x19:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1a:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1b:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1c:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1d:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x1f:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x20:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x21:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x22:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x23:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x24:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x25:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x26:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x27:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x28:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x29:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2a:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2b:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2c:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2d:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2e:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000003:0x2f:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x1:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x2:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x3:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20001:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20002:0x0]: rc = 0
Lustre: 740032:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0001: backup index [0x200000005:0x20003:0x0]: rc = 0
LDISKFS-fs (dm-3): unmounting filesystem 3ff6f387-7b80-4bdd-9a21-b667239432bd.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds4
Lustre: Failing over lustre-MDT0003
Lustre: *** cfs_fail_loc=199, val=0***
Lustre: Skipped 46 previous similar messages
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000001:0x3:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x3:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x4:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x5:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x6:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x7:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x8:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xd:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xe:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0xf:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x10:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x11:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x12:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x13:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x14:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x15:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x16:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x17:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x18:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x19:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1a:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1b:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1c:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1d:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x1f:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x20:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x21:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x22:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x23:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x24:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x25:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x26:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x27:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x28:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x29:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2a:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2b:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2c:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2d:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2e:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000003:0x2f:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x1:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x2:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x3:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20001:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20002:0x0]: rc = 0
Lustre: 740503:0:(scrub.c:1100:lustre_index_backup()) lustre-MDT0003: backup index [0x200000005:0x20003:0x0]: rc = 0
LDISKFS-fs (dm-4): unmounting filesystem cd055788-3a83-4097-bde7-f1e4c23dc312.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft -o user_xattr /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 3ff6f387-7b80-4bdd-9a21-b667239432bd r/w with ordered data mode. Quota mode: journalled.
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2020000-MDT0001' with [0x200000005:0x20003:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x20000-MDT0001' with [0x200000005:0x20001:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1020000-MDT0001' with [0x200000005:0x20002:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0
Lustre: 741786:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm8.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote -o user_xattr /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem cd055788-3a83-4097-bde7-f1e4c23dc312 r/w with ordered data mode. Quota mode: journalled.
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace' with [0x200000003:0xd:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2010000-MDT0003' with [0x200000005:0x3:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x10000-MDT0003' with [0x200000005:0x1:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1010000-MDT0003' with [0x200000005:0x2:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x20000' with [0x200000003:0x6:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1020000' with [0x200000003:0x7:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2010000' with [0x200000003:0x5:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x10000' with [0x200000003:0x3:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2020000-MDT0003' with [0x200000005:0x20003:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x20000-MDT0003' with [0x200000005:0x20001:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1010000' with [0x200000003:0x4:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x1020000-MDT0003' with [0x200000005:0x20002:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index '0x2020000' with [0x200000003:0x8:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_13' with [0x200000003:0x2c:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_04' with [0x200000003:0x23:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_04' with [0x200000003:0x12:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_02' with [0x200000003:0x10:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_12' with [0x200000003:0x2b:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_00' with [0x200000003:0xe:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_13' with [0x200000003:0x1b:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_09' with [0x200000003:0x28:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_09' with [0x200000003:0x17:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_10' with [0x200000003:0x29:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_05' with [0x200000003:0x24:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_11' with [0x200000003:0x19:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_14' with [0x200000003:0x2d:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_15' with [0x200000003:0x1d:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_02' with [0x200000003:0x21:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_01' with [0x200000003:0x20:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_03' with [0x200000003:0x22:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_03' with [0x200000003:0x11:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_15' with [0x200000003:0x2e:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_12' with [0x200000003:0x1a:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_00' with [0x200000003:0x1f:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_07' with [0x200000003:0x26:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_07' with [0x200000003:0x15:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_10' with [0x200000003:0x18:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_08' with [0x200000003:0x16:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_11' with [0x200000003:0x2a:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_14' with [0x200000003:0x1c:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_05' with [0x200000003:0x13:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_06' with [0x200000003:0x14:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_06' with [0x200000003:0x25:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_namespace_01' with [0x200000003:0xf:0x0]: rc = 0
Lustre: 743776:0:(osd_scrub.c:1862:osd_index_restore()) lustre-MDT0003: restore index 'lfsck_layout_08' with [0x200000003:0x27:0x0]: rc = 0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm9.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'

Link to test (2026-08-17 04:42)

sanity-pcc test 2a: Test multi open when creating

general protection fault, probably for non-canonical address 0x8bda9b69eaffc5df: 0000 [#1] SMP PTI
CPU: 0 UID: 0 PID: 99002 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffce4d478ffe78 EFLAGS: 00010246
RAX: ffff8dc184fec0a0 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8dc184fecb10 RSI: 0000000000000000 RDI: 8bda9b69eaffc5c7
RBP: ffff8dc1aaab9400 R08: 0000000000000040 R09: 0000000000000002
R10: 0000000000000000 R11: ffff8dc180f11b90 R12: 0000000000000000
R13: 00000171b79e4252 R14: ffffffffc13fc050 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ffff8dc1fbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f490c0ea260 CR3: 0000000077022004 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel kvm mlx5_ib irqbypass rapl iTCO_wdt ib_uverbs iTCO_vendor_support pcspkr i2c_i801 macsec mlx5_fwctl fwctl i2c_smbus virtio_balloon lpc_ich joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core ahci libahci mlxfw libata virtio_net tls crct10dif_pclmul crc32_pclmul virtio_blk crc32c_intel ghash_clmulni_intel psample net_failover pci_hyperv_intf failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod ib_ipoib ib_cm ib_core nfnetlink

Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
Lustre: Skipped 18 previous similar messages
LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8dc1fb5a2000 x1873493963218688/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.29.73@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem fe4e1c8b-b110-4c63-a8f4-3c2fc2327a0f.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0000: not available for connect from 10.240.29.69@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 50 previous similar messages
LustreError: 83390:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.29.73@tcp arrived at 1786704310 with bad export cookie 5758006088113629446
LustreError: 83390:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) Skipped 5 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: MGC10.240.29.72@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8dc19f69dd40 x1873493963230464/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LustreError: 11434:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
LDISKFS-fs (dm-4): unmounting filesystem 5d0ea3fd-b039-4036-973a-aa6b6fd8397c.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: tune2fs -O project /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: tune2fs -O project /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem fe4e1c8b-b110-4c63-a8f4-3c2fc2327a0f r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/l
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm26.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm27.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-146vm27.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 5d0ea3fd-b039-4036-973a-aa6b6fd8397c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/l

Link to test (2026-08-14 11:42)

conf-sanity test 53a: check OSS thread count params

general protection fault, probably for non-canonical address 0xac864f24560509f4: 0000 [#1] PREEMPT SMP PTI
CPU: 0 UID: 0 PID: 849745 Comm: lraft_0003 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffafb946e9fe90 EFLAGS: 00010246
RAX: ffff8a59a872ce90 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8a59a872c0d0 RSI: 0000000000000000 RDI: ac864f24560509dc
RBP: ffff8a59a18eee00 R08: 0000000000000040 R09: 0000000000000004
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 00001c4122860736 R14: ffff8a59b90b4000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8a59fbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005643865f0ce0 CR3: 0000000103c80006 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xd2/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel ib_umad(OE) kvm sunrpc rapl virtio_net iTCO_wdt iTCO_vendor_support net_failover mlx5_ib(OE) i2c_i801 ib_uverbs(OE) failover mlx5_fwctl(OE) fwctl(OE) ib_core(OE) virtio_balloon i2c_smbus pcspkr macsec lpc_ich joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core(OE) mlxfw(OE) tls ahci crct10dif_pclmul libahci psample crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_blk mlxdevm(OE)
mlx_compat(OE) pci_hyperv_intf serio_raw
Unloaded tainted modules: lzstd(OE):2 llz4hc(OE):2 llz4(OE):2 libcfs(OE):3 lnet(OE):3 obdclass(OE):3 ko2iblnd(OE):3 ldiskfs(OE):3 ptlrpc(OE):3 lquota(OE):3 ptlrpc_gss(OE):2 lraft(OE):3 mgc(OE):3 osc(OE):3 fld(OE):3 lov(OE):3 fid(OE):3 lfsck(OE):3 lod(OE):3 osd_ldiskfs(OE):3 mdt(OE):3 lmv(OE):3 mdc(OE):3 lustre(OE):3 osp(OE):3 obdecho(OE):2 mdd(OE):3 [last unloaded: libcfs(OE)]

Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Autotest: Test running for 510 minutes (lustre-b_es7_0_full-dne-o2ib-part-3_315.15)
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2
LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Key type lgssc unregistered
LNet: 828826:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 172.16.12.13@o2ib
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn55
LNet: Using FastReg for registration
LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0003: Not available for connect from 172.16.12.12@o2ib (not set up)
Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 172.16.12.11@o2ib failed: rc = -107
Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 172.16.12.11@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 172.16.12.12@o2ib failed: rc = -107
Lustre: lustre-MDT0000-lwp-MDT0003: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2
LustreError: MGC172.16.12.12@o2ib: Connection to MGS (at 172.16.12.12@o2ib) was lost; in progress operations using this service will fail
Lustre: lustre-MDT0001: Not available for connect from 172.16.12.12@o2ib (stopping)
Lustre: Skipped 1 previous similar message
LustreError: lustre-MDT0001-osp-MDT0003: operation mds_statfs to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: 830049:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ff32e4cc x1873268767260160/t0(0) o1203->lustre-MDT0000-osp-MDT0001@172.16.12.12@o2ib:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295
LustreError: 830050:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000601e40ee x1873268767260544/t0(0) o1203->lustre-MDT0000-osp-MDT0001@172.16.12.12@o2ib:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295
LustreError: 830050:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
Lustre: Skipped 1 previous similar message
LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Key type lgssc unregistered
LNet: 837576:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 172.16.12.13@o2ib
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn55
LNet: Using FastReg for registration
LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180]
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0001: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0001: Not available for connect from 172.16.12.12@o2ib (not set up)
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 22 previous similar messages
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm9.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: lustre-OST0000-osc-MDT0003: Connection to lustre-OST0000 (at 172.16.12.11@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 172.16.12.11@o2ib failed: rc = -107
Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Autotest: Test running for 515 minutes (lustre-b_es7_0_full-dne-o2ib-part-3_315.15)
LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 172.16.12.12@o2ib failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds2
LustreError: MGC172.16.12.12@o2ib: Connection to MGS (at 172.16.12.12@o2ib) was lost; in progress operations using this service will fail
Lustre: lustre-MDT0001-osp-MDT0003: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
Lustre: Skipped 1 previous similar message
LDISKFS-fs (dm-3): unmounting filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca.
Lustre: server umount lustre-MDT0001 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 5 previous similar messages
Lustre: lustre-MDT0002-osp-MDT0003: Connection to lustre-MDT0002 (at 172.16.12.12@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds4
LDISKFS-fs (dm-4): unmounting filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c.
Lustre: server umount lustre-MDT0003 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing unload_modules_local
LNet: 845472:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 172.16.12.13@o2ib
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm11.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm11.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm10.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn55
LNet: Using FastReg for registration
LNet: Added LNI 172.16.12.13@o2ib [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 66e41de2-3c75-4b3d-8960-756c5ef9b0ca r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-140vm13.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
LustreError: lustre-MDT0003: not available for connect from 172.16.12.12@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
LustreError: lustre-MDT0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 2 previous similar messages
Lustre: DEBUG MARKER: onyx-140vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 36ec645e-a4d6-4cd0-935e-4e89da892e2c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-08-12 01:56)

conf-sanity test 26: MDT startup failure cleans LOV (should return errs)

general protection fault, probably for non-canonical address 0x97aec02a62ccb9e7: 0000 [#1] SMP PTI
CPU: 0 UID: 0 PID: 500943 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1_lustre.ddn1.x86_64 #1 PREEMPT(voluntary)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffffcefb8ca57e78 EFLAGS: 00010246
RAX: ffff8b7ac9a4e428 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8b7ac9a4e838 RSI: 0000000000000000 RDI: 97aec02a62ccb9cf
RBP: ffff8b7ac1ec0400 R08: 0000000000000040 R09: 0000000000000004
R10: ffffffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 000004b9e50688e1 R14: ffffffffc17e9050 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8b7b3bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c4d5061f50 CR3: 000000001e622003 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? __pfx_lraft_timerd+0x10/0x10 [lraft]
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x155/0x350 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs intel_rapl_msr intel_rapl_common rfkill intel_uncore_frequency_common mlx5_ib kvm_intel ib_uverbs kvm macsec mlx5_fwctl irqbypass fwctl rapl iTCO_wdt i2c_i801 iTCO_vendor_support lpc_ich pcspkr virtio_balloon i2c_smbus joydev fuse loop vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 mlx5_core virtio_net ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata mlxfw ghash_clmulni_intel net_failover failover tls virtio_blk psample pci_hyperv_intf serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod ib_ipoib ib_cm ib_core nfnetlink [last unloaded: dm_flakey]

Lustre: DEBUG MARKER: lsmod | grep -q lustre || modprobe lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000135
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-4): mounted filesystem 296314a3-b64c-41ae-a550-f4948704dd1a r/w with ordered data mode. Quota mode: journalled.
Lustre: *** cfs_fail_loc=135, val=0***
LustreError: 497090:0:(obd_config.c:801:class_setup()) setup lustre-MDT0000 failed (-2)
LustreError: 497090:0:(obd_config.c:2032:class_config_llog_handler()) MGC10.240.26.60@tcp: cfg command failed: rc = -2
Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f
LustreError: MGC10.240.26.60@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info
LustreError: 497051:0:(tgt_mount.c:1843:server_start_targets()) failed to start server lustre-MDT0000: -2
LustreError: 497051:0:(tgt_mount.c:2587:server_fill_super()) Unable to start targets: -2
LustreError: 497051:0:(obd_config.c:854:class_cleanup()) Device 5 not setup
Lustre: server umount lustre-MDT0000 complete
LustreError: 497051:0:(super25.c:206:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2
LDISKFS-fs (dm-4): unmounting filesystem 296314a3-b64c-41ae-a550-f4948704dd1a.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-4): mounted filesystem 296314a3-b64c-41ae-a550-f4948704dd1a r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-141vm32.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-141vm33.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-141vm33.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-5): mounted filesystem 1f49507d-b08c-47f6-8b38-f16a412a333a r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-08-10 09:51)

sanity-sec test 27ac: test nodemap llog and IAM fileset compatibility

general protection fault, probably for non-canonical address 0x8859eafdb3f5b2d6: 0000 [#1] PREEMPT SMP PTI
CPU: 1 UID: 0 PID: 596278 Comm: lraft_0002 Kdump: loaded Tainted: G OE ------- --- 6.12.0-55.43.1.el10_0_lustre.ddn1.x86_64 #1
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_is_active+0x9/0x20 [lraft]
Code: cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 <0f> b6 47 18 c0 e8 03 83 e0 01 83 f0 01 c3 cc cc cc cc 0f 1f 44 00
RSP: 0018:ffff99c3c5bf3e90 EFLAGS: 00010246
RAX: ffff8a158ded4b80 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff8a158ded4cd8 RSI: 0000000000000000 RDI: 8859eafdb3f5b2be
RBP: ffff8a158544ac00 R08: 0000000000000040 R09: 0000000000000002
R10: 0000000000000000 R11: ffff8a15fbc4d0d0 R12: 0000000000000000
R13: 000007f0e1b082c5 R14: ffff8a15b4db4000 R15: 0000000000000007
FS: 0000000000000000(0000) GS:ffff8a15fbd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000556ac7ccef70 CR3: 0000000102716003 CR4: 0000000000372ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1b0/0x2f0
? show_trace_log_lvl+0x1b0/0x2f0
? raft_get_num_voting_nodes+0x34/0x70 [lraft]
? __die_body.cold+0x8/0x12
? die_addr+0x3c/0x60
? exc_general_protection+0x178/0x410
? asm_exc_general_protection+0x26/0x30
? raft_node_is_active+0x9/0x20 [lraft]
raft_get_num_voting_nodes+0x34/0x70 [lraft]
lraft_timerd+0x153/0x320 [lraft]
? __pfx_lraft_timerd+0x10/0x10 [lraft]
kthread+0xd2/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lzstd(OE) llz4hc(OE) llz4(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace nfs_localio netfs rfkill mlx5_ib ib_uverbs macsec intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel mlx5_core sunrpc kvm mlxfw psample iTCO_wdt rapl iTCO_vendor_support tls virtio_net net_failover i2c_i801 pci_hyperv_intf failover lpc_ich pcspkr virtio_balloon i2c_smbus joydev fuse loop dm_mod nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk serio_raw ib_ipoib ib_cm ib_core
Unloaded tainted modules: lzstd(OE):3 llz4hc(OE):3 llz4(OE):3 ptlrpc_gss(OE):3 obdecho(OE):3 libcfs(OE):4 lnet(OE):4 ksocklnd(OE):4 obdclass(OE):4 ldiskfs(OE):4 ptlrpc(OE):4 lquota(OE):4 lraft(OE):4 mgc(OE):4 mgs(OE):4 osc(OE):4 fld(OE):4 lov(OE):4 fid(OE):4 lfsck(OE):4 lod(OE):4 osd_ldiskfs(OE):4 mdt(OE):4 lmv(OE):4 mdc(OE):4 lustre(OE):4 osp(OE):4 mdd(OE):4 [last unloaded: lzstd(OE)]

Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add test27ac_nm
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset /iam_fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_del test27ac_nm ; /usr/sbin/lctl nodemap_add test27ac_nm
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param nodemap.test27ac_nm.fileset=/llog_fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P nodemap.test27ac_nm.fileset=/llog_fileset
Lustre: Setting parameter general.nodemap.test27ac_nm.fileset=/llog_fileset in log params
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping)
Lustre: Skipped 18 previous similar messages
Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15aa835d40 x1872805408788864/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem b785b3f1-096b-49a6-83f7-964da059cf10.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: 555405:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.70@tcp arrived at 1786047055 with bad export cookie 13200158399717358067
LustreError: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 10.240.27.70@tcp failed: rc = -107
LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 5 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: 555405:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047063 with bad export cookie 13200158399717357633
LustreError: MGC10.240.27.69@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15aaa0d6c0 x1872805408799232/t0(0) o1202->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 256/240 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LustreError: 517885:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LDISKFS-fs (dm-4): unmounting filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Key type lgssc unregistered
LNet: 572521:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNetError: 572521:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found
LNet: Removed LNI 10.240.27.69@tcp
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn55_2_g66cf7cf
LNet: Added LNI 10.240.27.69@tcp [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem b785b3f1-096b-49a6-83f7-964da059cf10 r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0002.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.raft
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: 581109:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: 581225:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Autotest: Test running for 135 minutes (lustre-b_es-reviews_review-dne-exa7-part-1_29794.157)
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset | awk '/primary/ { if ($3 == "/llog_fileset") print $3 }'
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset /iam_fileset
Lustre: Disabling parameter general.nodemap.test27ac_nm.fileset= in log params
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param nodemap.test27ac_nm.fileset=/llog_fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name test27ac_nm --fileset clear
Lustre: 585940:0:(mgs_llog.c:1439:mgs_modify_param()) MGS: modify general/nodemap.test27ac_nm.fileset= (mode = 1) failed: rc = -2
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.test27ac_nm.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
Lustre: lustre-MDT0000: Not available for connect from 10.240.27.70@tcp (stopping)
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a158d6eb400 x1872806155674368/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping)
Lustre: Skipped 6 previous similar messages
LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15fb58ecc0 x1872806155675392/t0(0) o1203->lustre-MDT0001-osp-MDT0000@10.240.27.70@tcp:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 575123:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Lustre: lustre-MDT0000: Not available for connect from 10.240.27.70@tcp (stopping)
Lustre: Skipped 6 previous similar messages
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15ab154340 x1872806155678848/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping)
Lustre: Skipped 5 previous similar messages
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15ab156d80 x1872806155680640/t0(0) o1203->lustre-MDT0002-osp-MDT0000@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0000.0' uid:0 gid:0 projid:4294967295
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Lustre: lustre-MDT0000: Not available for connect from 10.240.30.224@tcp (stopping)
Lustre: Skipped 13 previous similar messages
Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
LDISKFS-fs (dm-3): unmounting filesystem b785b3f1-096b-49a6-83f7-964da059cf10.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
LustreError: 576098:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 10.240.27.70@tcp arrived at 1786047274 with bad export cookie 774761076203025042
LustreError: lustre-MDT0001-osp-MDT0002: operation raft_appendentries to node 10.240.27.70@tcp failed: rc = -107
Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 10.240.27.70@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3
LustreError: 576099:0:(ldlm_lockd.c:2574:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786047282 with bad export cookie 774761076203024601
LustreError: MGC10.240.27.69@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a15fb4e6000 x1872806155692416/t0(0) o1203->lustre-MDT0000-osp-MDT0002@0@lo:24/4 lens 264/256 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lraft_0002.0' uid:0 gid:0 projid:4294967295
LustreError: 575122:0:(client.c:1304:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: lustre-MDT0000: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 24 previous similar messages
Lustre: lustre-MDT0002: Not available for connect from 10.240.27.70@tcp (stopping)
Lustre: Skipped 7 previous similar messages
LDISKFS-fs (dm-4): unmounting filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing unload_modules_local
Key type lgssc unregistered
LNet: 589984:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNetError: 589984:0:(acceptor.c:270:lnet_acceptor_remove_socket()) Interface eth0 not found
LNet: Removed LNI 10.240.27.69@tcp
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm58.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-145vm20.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-180vm58.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn55_2_g66cf7cf
LNet: Added LNI 10.240.27.69@tcp [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem b785b3f1-096b-49a6-83f7-964da059cf10 r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm16.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
LustreError: lustre-MDT0002: not available for connect from 10.240.27.70@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm17.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
LustreError: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 1 previous similar message
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov,raft /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-4): mounted filesystem 7b153bfb-3f9a-46c3-8947-512564b85dd6 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u

Link to test (2026-08-06 18:27)

Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below):
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below):
Limit to a test: (Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Editing crashreport #75243

Added fields:

Failures list (last 100):