Editing crashreport #73857

ReasonCrashing FunctionWhere to cut BacktraceReports Count
general protection fault, raft_node_get_idraft_get_node
lraft_appendentries_interpret
ptlrpc_check_set
ptlrpcd_check
ptlrpcd
kthread
ret_from_fork
3

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
conf-sanity test 103: rename filesystem name
general protection fault, probably for non-canonical address 0x543e60a8cc05f3be: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 546665 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:raft_node_get_id+0x5/0x10 [lraft]
Code: 83 e0 01 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <8b> 47 1c c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
RSP: 0018:ffffc3c9c0993cd0 EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffffa0fd02d356d0 RSI: 0000000000000000 RDI: 543e60a8cc05f3a2
RBP: ffffa0fd2f8e0a00 R08: ffffa0fd293719c0 R09: 0000000000000020
R10: ffffa0fd293719c0 R11: 0000000000000020 R12: 0000000000000000
R13: 0000000000000008 R14: ffffa0fd2e077a80 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffffa0fd3bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff8652a34e0 CR3: 000000011fa10005 CR4: 0000000000370ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? raft_get_node+0x3c/0x70 [lraft]
? __die_body.cold+0x8/0xd
? die_addr+0x39/0x60
? exc_general_protection+0x1c8/0x420
? asm_exc_general_protection+0x22/0x30
? raft_node_get_id+0x5/0x10 [lraft]
raft_get_node+0x3c/0x70 [lraft]
lraft_appendentries_interpret+0xc0/0x280 [lraft]
ptlrpc_check_set+0x3dc/0x1f30 [ptlrpc]
ptlrpcd_check+0x431/0x5c0 [ptlrpc]
ptlrpcd+0x20d/0x4a0 [ptlrpc]
? __pfx_woken_wake_function+0x10/0x10
? __pfx_ptlrpcd+0x10/0x10 [ptlrpc]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel kvm rapl rfkill pcspkr virtio_balloon i2c_piix4 joydev sunrpc drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net net_failover failover ghash_clmulni_intel virtio_blk serio_raw [last unloaded: lzstd(OE)]
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm4.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-146vm4.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem b0e1dd28-d445-43c1-928d-5a4334dfea43 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-146vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-146vm12.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-146vm4.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-146vm4.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem abca1246-84a4-4c81-8a62-a6420af8fad3 r/w with ordered data mode. Quota mode: journalled.
Link to test
conf-sanity test 34a: umount with opened file should be fail
general protection fault, probably for non-canonical address 0x1294cb9ba92facd5: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 9305 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014
RIP: 0010:raft_node_get_id+0x5/0x10 [lraft]
Code: 83 e0 01 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <8b> 47 1c c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
RSP: 0018:ffffab71c08a3cd0 EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000000
RDX: ffff9162ea776160 RSI: 0000000000000000 RDI: 1294cb9ba92facb9
RBP: ffff9162f49f4400 R08: ffff9162f4fcebc0 R09: 0000000000000020
R10: ffff9162f4fcebc0 R11: 0000000000000020 R12: 0000000000000000
R13: 0000000000000008 R14: ffff9162dfdde080 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9162fbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000561dae05205c CR3: 0000000119c6e004 CR4: 0000000000370ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? raft_get_node+0x3c/0x70 [lraft]
? __die_body.cold+0x8/0xd
? die_addr+0x39/0x60
? exc_general_protection+0x1c8/0x420
? asm_exc_general_protection+0x22/0x30
? raft_node_get_id+0x5/0x10 [lraft]
raft_get_node+0x3c/0x70 [lraft]
lraft_appendentries_interpret+0xc0/0x280 [lraft]
ptlrpc_check_set+0x3dc/0x1f30 [ptlrpc]
ptlrpcd_check+0x431/0x5c0 [ptlrpc]
ptlrpcd+0x20d/0x4a0 [ptlrpc]
? __pfx_woken_wake_function+0x10/0x10
? __pfx_ptlrpcd+0x10/0x10 [ptlrpc]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common kvm_intel kvm rapl virtio_balloon i2c_piix4 pcspkr joydev drm fuse ext4 mbcache jbd2 crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic virtio_net ata_piix ghash_clmulni_intel libata net_failover failover virtio_blk serio_raw
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm10.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-142vm10.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-3): mounted filesystem 71c8fe82-3148-43e9-9dce-d2df4a50010d r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-142vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-142vm11.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm10.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-142vm10.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft_nonvote /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-4): mounted filesystem 5d45b1c1-a560-43d1-baee-fbf8096c2423 r/w with ordered data mode. Quota mode: journalled.
Link to test
Module load
general protection fault, probably for non-canonical address 0xd5fe380863d5dde2: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 11483 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:raft_node_get_id+0x5/0x10 [lraft]
Code: 83 e0 01 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <8b> 47 1c c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
RSP: 0018:ffff9d32c09bbcd0 EFLAGS: 00010202
RAX: 0000000000000002 RBX: 0000000000000002 RCX: 0000000000000000
RDX: ffff8d80c4eacfe0 RSI: 0000000000000002 RDI: d5fe380863d5ddc6
RBP: ffff8d80e93e7000 R08: ffff8d80c604d9c0 R09: 0000000000000020
R10: ffff8d80c604d9c0 R11: 0000000000000020 R12: 0000000000000002
R13: 0000000000000010 R14: ffff8d80f3f1ba80 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8d80fbc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f636b44050 CR3: 00000001016ea005 CR4: 00000000003706f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? raft_get_node+0x3c/0x70 [lraft]
? __die_body.cold+0x8/0xd
? die_addr+0x39/0x60
? exc_general_protection+0x1c8/0x420
? asm_exc_general_protection+0x22/0x30
? raft_node_get_id+0x5/0x10 [lraft]
raft_get_node+0x3c/0x70 [lraft]
lraft_appendentries_interpret+0xc0/0x280 [lraft]
ptlrpc_check_set+0x3dc/0x1f30 [ptlrpc]
ptlrpcd_check+0x431/0x5c0 [ptlrpc]
ptlrpcd+0x20d/0x4a0 [ptlrpc]
? __pfx_woken_wake_function+0x10/0x10
? __pfx_ptlrpcd+0x10/0x10 [ptlrpc]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) lraft(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr i2c_piix4 virtio_balloon joydev drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-142vm13.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-142vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.20@tcp --fsname=lustre --mdt --index=1 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=786432 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt2
LDISKFS-fs (dm-0): mounted filesystem a03aba88-c5b3-4705-ba40-e5070fc30a7e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem a03aba88-c5b3-4705-ba40-e5070fc30a7e.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.20@tcp --fsname=lustre --mdt --index=3 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=786432 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt4
LDISKFS-fs (dm-1): mounted filesystem 749fe978-c449-4f18-8090-639cac616772 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 749fe978-c449-4f18-8090-639cac616772.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.20@tcp --fsname=lustre --mdt --index=5 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=786432 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt6
LDISKFS-fs (dm-2): mounted filesystem ece39630-1880-4ee5-a79f-1d22eec21c78 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-2): unmounting filesystem ece39630-1880-4ee5-a79f-1d22eec21c78.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.29.20@tcp --fsname=lustre --mdt --index=7 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=786432 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt8
LDISKFS-fs (dm-3): mounted filesystem 720a968f-4d07-4fae-b3fb-659678777f30 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 720a968f-4d07-4fae-b3fb-659678777f30.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds2_flakey --table "0 1974272 linear /dev/vg_Role_MDS/mdt2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov,raft /dev/mapper/mds2_flakey /mnt/lustre-mds2
LDISKFS-fs (dm-5): mounted filesystem a03aba88-c5b3-4705-ba40-e5070fc30a7e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-5): unmounting filesystem a03aba88-c5b3-4705-ba40-e5070fc30a7e.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn31_2_gac59d61
LNet: Added LNI 10.240.29.21@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-5): mounted filesystem a03aba88-c5b3-4705-ba40-e5070fc30a7e r/w with ordered data mode. Quota mode: journalled.
Lustre: 11461:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.29.20@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
Lustre: lustre-MDT0001: new disk, initializing
Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds4_flakey --table "0 1974272 linear /dev/vg_Role_MDS/mdt4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov,raft /dev/mapper/mds4_flakey /mnt/lustre-mds4
LDISKFS-fs (dm-6): mounted filesystem 749fe978-c449-4f18-8090-639cac616772 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-6): unmounting filesystem 749fe978-c449-4f18-8090-639cac616772.
LDISKFS-fs (dm-6): mounted filesystem 749fe978-c449-4f18-8090-639cac616772 r/w with ordered data mode. Quota mode: journalled.
Lustre: 13645:0:(mgc_request_server.c:552:mgc_llog_local_copy()) MGC10.240.29.20@tcp: no remote llog for lustre-sptlrpc, check MGS config
Lustre: srv-lustre-MDT0003: No data found on store. Initialize space.
Lustre: lustre-MDT0003: new disk, initializing
Lustre: lustre-MDT0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: cli-ctl-lustre-MDT0003: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3:mdt]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm14.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-142vm13.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt6 2>/dev/null
Link to test
Return to new crashes list