Editing crashreport #73996

ReasonCrashing FunctionWhere to cut BacktraceReports Count
watchdog: BUG: soft lockup - __pv_queued_spin_lock_slowpath_raw_spin_lock
lnet_peer_discovery
kthread
ret_from_fork
panic
watchdog_timer_fn
__hrtimer_run_queues
hrtimer_interrupt
__sysvec_apic_timer_interrupt
sysvec_apic_timer_interrupt
asm_sysvec_apic_timer_interrupt
2

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [lnet_discovery:10058]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common i2c_piix4 virtio_balloon pcspkr joydev drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw
CPU: 1 PID: 10058 Comm: lnet_discovery Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:__pv_queued_spin_lock_slowpath+0xf4/0x370
Code: 14 41 bd 01 00 00 00 41 be 00 01 00 00 3c 02 0f 94 c0 0f b6 c0 48 89 04 24 c6 43 14 00 ba 00 80 00 00 c6 45 01 01 eb 0b f3 90 <83> ea 01 0f 84 1c 02 00 00 0f b6 45 00 84 c0 75 ed 44 89 f0 f0 66
RSP: 0018:ffffb9b9c0807e60 EFLAGS: 00000206
RAX: 0000000000000003 RBX: ffff9d983fd34900 RCX: 0000000000000008
RDX: 0000000000006b0c RSI: 0000000000000003 RDI: ffff9d9786bb7cb4
RBP: ffff9d9786bb7cb4 R08: ffff9d983ffc0940 R09: 0000000000000000
R10: ffff9d983fd34900 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000080000
FS: 0000000000000000(0000) GS:ffff9d983fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe22133f6c CR3: 000000000314a006 CR4: 00000000001706f0
Call Trace:
<IRQ>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? _raw_spin_lock+0x25/0x30
? watchdog_timer_fn+0x1ad/0x210
? __pfx_watchdog_timer_fn+0x10/0x10
? __hrtimer_run_queues+0x112/0x2b0
? hrtimer_interrupt+0xfc/0x210
? __do_softirq+0x169/0x2a8
? __sysvec_apic_timer_interrupt+0x4e/0x100
? sysvec_apic_timer_interrupt+0x6d/0x90
</IRQ>
<TASK>
? asm_sysvec_apic_timer_interrupt+0x16/0x20
? __pv_queued_spin_lock_slowpath+0xf4/0x370
_raw_spin_lock+0x25/0x30
lnet_peer_discovery+0x699/0xb00 [lnet]
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_lnet_peer_discovery+0x10/0x10 [lnet]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Kernel panic - not syncing: softlockup: hung tasks
CPU: 1 PID: 10058 Comm: lnet_discovery Kdump: loaded Tainted: G OEL ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<IRQ>
dump_stack_lvl+0x34/0x48
panic+0x107/0x2bb
watchdog_timer_fn.cold+0xc/0x16
? __pfx_watchdog_timer_fn+0x10/0x10
__hrtimer_run_queues+0x112/0x2b0
hrtimer_interrupt+0xfc/0x210
? __do_softirq+0x169/0x2a8
__sysvec_apic_timer_interrupt+0x4e/0x100
sysvec_apic_timer_interrupt+0x6d/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20
RIP: 0010:__pv_queued_spin_lock_slowpath+0xf4/0x370
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-92vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem 782c0a7e-a948-4158-bd15-26c51e39e15e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 782c0a7e-a948-4158-bd15-26c51e39e15e.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.43.165@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem c4a2128f-b89f-4d13-957a-e4c784c5fc2e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem c4a2128f-b89f-4d13-957a-e4c784c5fc2e.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 782c0a7e-a948-4158-bd15-26c51e39e15e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 782c0a7e-a948-4158-bd15-26c51e39e15e.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.43.165@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 782c0a7e-a948-4158-bd15-26c51e39e15e r/w with ordered data mode. Quota mode: journalled.
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [lnet_discovery:10054]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs sunrpc rfkill intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net net_failover ghash_clmulni_intel virtio_blk failover serio_raw
CPU: 0 PID: 10054 Comm: lnet_discovery Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:__pv_queued_spin_lock_slowpath+0xf4/0x370
Code: 14 41 bd 01 00 00 00 41 be 00 01 00 00 3c 02 0f 94 c0 0f b6 c0 48 89 04 24 c6 43 14 00 ba 00 80 00 00 c6 45 01 01 eb 0b f3 90 <83> ea 01 0f 84 1c 02 00 00 0f b6 45 00 84 c0 75 ed 44 89 f0 f0 66
RSP: 0018:ffff97820089fe60 EFLAGS: 00000206
RAX: 0000000000000003 RBX: ffff88f43fc34900 RCX: 0000000000000008
RDX: 0000000000000a98 RSI: 0000000000000003 RDI: ffff88f3851a00b4
RBP: ffff88f3851a00b4 R08: ffff88f43ffc0180 R09: 0000000000000000
R10: ffff88f43fc34900 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000040000
FS: 0000000000000000(0000) GS:ffff88f43fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005582f8eeefc0 CR3: 000000000a2be003 CR4: 00000000000606f0
Call Trace:
<IRQ>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? _raw_spin_lock+0x25/0x30
? watchdog_timer_fn+0x1ad/0x210
? __pfx_watchdog_timer_fn+0x10/0x10
? __hrtimer_run_queues+0x112/0x2b0
? hrtimer_interrupt+0xfc/0x210
? __do_softirq+0x169/0x2a8
? __sysvec_apic_timer_interrupt+0x4e/0x100
? sysvec_apic_timer_interrupt+0x6d/0x90
</IRQ>
<TASK>
? asm_sysvec_apic_timer_interrupt+0x16/0x20
? __pv_queued_spin_lock_slowpath+0xf4/0x370
_raw_spin_lock+0x25/0x30
lnet_peer_discovery+0x699/0xb00 [lnet]
? __pfx_autoremove_wake_function+0x10/0x10
watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [llog_process_th:10245]
? __pfx_lnet_peer_discovery+0x10/0x10 [lnet]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey
kthread+0xe0/0x100
dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs sunrpc rfkill intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev drm fuse ext4 mbcache jbd2 ata_generic
? __pfx_kthread+0x10/0x10
ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net net_failover ghash_clmulni_intel virtio_blk failover serio_raw
ret_from_fork+0x2c/0x50
CPU: 1 PID: 10245 Comm: llog_process_th Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
</TASK>
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Kernel panic - not syncing: softlockup: hung tasks
CPU: 0 PID: 10054 Comm: lnet_discovery Kdump: loaded Tainted: G OEL ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<IRQ>
dump_stack_lvl+0x34/0x48
panic+0x107/0x2bb
watchdog_timer_fn.cold+0xc/0x16
? __pfx_watchdog_timer_fn+0x10/0x10
__hrtimer_run_queues+0x112/0x2b0
hrtimer_interrupt+0xfc/0x210
? __do_softirq+0x169/0x2a8
__sysvec_apic_timer_interrupt+0x4e/0x100
sysvec_apic_timer_interrupt+0x6d/0x90
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x16/0x20
RIP: 0010:__pv_queued_spin_lock_slowpath+0xf4/0x370
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-130vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-130vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-130vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-130vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-130vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-130vm5.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-130vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-130vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem 33fa351a-0ad0-4feb-8152-196f0a3bbe85 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-0): unmounting filesystem 33fa351a-0ad0-4feb-8152-196f0a3bbe85.
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.45.48@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem 3667183a-0067-4de9-ba9e-fe0743a0572c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-1): unmounting filesystem 3667183a-0067-4de9-ba9e-fe0743a0572c.
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 33fa351a-0ad0-4feb-8152-196f0a3bbe85 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 33fa351a-0ad0-4feb-8152-196f0a3bbe85.
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.45.48@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem 33fa351a-0ad0-4feb-8152-196f0a3bbe85 r/w with ordered data mode. Quota mode: journalled.
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Return to new crashes list