Editing crashreport #73995

ReasonCrashing FunctionWhere to cut BacktraceReports Count
watchdog: BUG: soft lockup - native_safe_haltkvm_wait
__pv_queued_spin_lock_slowpath
_raw_spin_lock
lnet_peer_discovery
kthread
ret_from_fork
panic
watchdog_timer_fn
__hrtimer_run_queues
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
2

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lnet_discovery:12669]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs intel_rapl_msr lockd grace intel_rapl_common fscache crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev virtio_balloon pcspkr i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net net_failover serio_raw failover virtio_blk
watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [llog_process_th:13038]
CPU: 0 PID: 12669 Comm: lnet_discovery Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Modules linked in:
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
mdd(OE)
RIP: 0010:native_safe_halt+0xe/0x20
lod(OE)
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
mdt(OE)
RSP: 0018:ffffa09280edfe00 EFLAGS: 00000246
lfsck(OE)
ORIG_RAX: ffffffffffffff13
mgs(OE)
RAX: 0000000000000003 RBX: ffff899384fa6cb4 RCX: 0000000000000008
mgc(OE)
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff899384fa6cb4
osd_ldiskfs(OE)
RBP: ffff89943cc345c0 R08: 0000000000000008 R09: 000000000000005c
lquota(OE)
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
lustre(OE)
R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000040000
mdc(OE)
FS: 0000000000000000(0000) GS:ffff89943cc00000(0000) knlGS:0000000000000000
lov(OE)
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
osc(OE)
CR2: 000055d26d428528 CR3: 00000000ac410005 CR4: 00000000001706f0
lmv(OE)
Call Trace:
fid(OE)
<IRQ>
fld(OE)
? watchdog_timer_fn.cold.10+0x46/0x9e
ksocklnd(OE)
? watchdog+0x30/0x30
ptlrpc(OE)
? __hrtimer_run_queues+0x101/0x280
obdclass(OE)
? hrtimer_interrupt+0x100/0x220
lnet(OE)
? smp_apic_timer_interrupt+0x6a/0x130
ldiskfs(OE)
? apic_timer_interrupt+0xf/0x20
libcfs(OE)
</IRQ>
dm_flakey
? native_safe_halt+0xe/0x20
dm_mod
kvm_wait+0x58/0x60
rpcsec_gss_krb5
__pv_queued_spin_lock_slowpath+0x268/0x2a0
auth_rpcgss
_raw_spin_lock+0x1e/0x30
nfsv4
lnet_peer_discovery+0xccc/0x1b90 [lnet]
dns_resolver
? finish_task_switch+0x86/0x2f0
nfs
? finish_wait+0x80/0x80
intel_rapl_msr
? lnet_peer_merge_data+0x10c0/0x10c0 [lnet]
lockd
kthread+0x134/0x150
grace
? set_kthread_struct+0x50/0x50
intel_rapl_common
ret_from_fork+0x35/0x40
fscache
Kernel panic - not syncing: softlockup: hung tasks
crct10dif_pclmul
CPU: 0 PID: 12669 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
crc32_pclmul
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
ghash_clmulni_intel
Call Trace:
joydev
<IRQ>
virtio_balloon
dump_stack+0x41/0x60
pcspkr
panic+0xe7/0x2ac
i2c_piix4
? syscall_return_via_sysret+0x6e/0x94
sunrpc
watchdog_timer_fn.cold.10+0x85/0x9e
ext4
? watchdog+0x30/0x30
mbcache
__hrtimer_run_queues+0x101/0x280
jbd2
hrtimer_interrupt+0x100/0x220
ata_generic
smp_apic_timer_interrupt+0x6a/0x130
ata_piix
apic_timer_interrupt+0xf/0x20
libata
</IRQ>
crc32c_intel
RIP: 0010:native_safe_halt+0xe/0x20
virtio_net
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-71vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm9.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-71vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.26.4@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.26.4@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lnet_discovery:12660]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr nfs intel_rapl_common lockd grace fscache crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 sunrpc ext4 mbcache ata_generic jbd2 ata_piix libata crc32c_intel serio_raw virtio_net net_failover failover virtio_blk
watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [llog_process_th:13029]
CPU: 0 PID: 12660 Comm: lnet_discovery Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Modules linked in:
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
mdd(OE)
RIP: 0010:native_safe_halt+0xe/0x20
lod(OE)
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
mdt(OE)
RSP: 0018:ffffb2a240e8be00 EFLAGS: 00000246
lfsck(OE)
ORIG_RAX: ffffffffffffff13
mgs(OE)
RAX: 0000000000000003 RBX: ffffa09d471360b4 RCX: 0000000000000008
mgc(OE)
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffa09d471360b4
osd_ldiskfs(OE)
RBP: ffffa09d7bc345c0 R08: 0000000000000008 R09: 0000000000000054
lquota(OE)
R10: 0000000000000000 R11: 0000000000000012 R12: 0000000000000000
lustre(OE)
R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000040000
mdc(OE)
FS: 0000000000000000(0000) GS:ffffa09d7bc00000(0000) knlGS:0000000000000000
lov(OE)
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
osc(OE)
CR2: 0000561a72712718 CR3: 000000002c610003 CR4: 00000000003706f0
lmv(OE)
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
fid(OE)
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
fld(OE)
Call Trace:
ksocklnd(OE)
<IRQ>
ptlrpc(OE)
? watchdog_timer_fn.cold.10+0x46/0x9e
obdclass(OE)
? watchdog+0x30/0x30
lnet(OE)
? __hrtimer_run_queues+0x101/0x280
ldiskfs(OE)
? hrtimer_interrupt+0x100/0x220
libcfs(OE)
? smp_apic_timer_interrupt+0x6a/0x130
dm_flakey
? apic_timer_interrupt+0xf/0x20
dm_mod
</IRQ>
rpcsec_gss_krb5
? native_safe_halt+0xe/0x20
auth_rpcgss
kvm_wait+0x58/0x60
nfsv4
__pv_queued_spin_lock_slowpath+0x268/0x2a0
dns_resolver
_raw_spin_lock+0x1e/0x30
intel_rapl_msr
lnet_peer_discovery+0xccc/0x1b90 [lnet]
nfs
? finish_task_switch+0x86/0x2f0
intel_rapl_common
? finish_wait+0x80/0x80
lockd
? lnet_peer_merge_data+0x10c0/0x10c0 [lnet]
grace
kthread+0x134/0x150
fscache
? set_kthread_struct+0x50/0x50
crct10dif_pclmul
ret_from_fork+0x35/0x40
crc32_pclmul
Kernel panic - not syncing: softlockup: hung tasks
ghash_clmulni_intel
CPU: 0 PID: 12660 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
joydev
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
pcspkr
Call Trace:
virtio_balloon
<IRQ>
i2c_piix4
dump_stack+0x41/0x60
sunrpc
panic+0xe7/0x2ac
ext4
? syscall_return_via_sysret+0x6e/0x94
mbcache
watchdog_timer_fn.cold.10+0x85/0x9e
ata_generic
? watchdog+0x30/0x30
jbd2
__hrtimer_run_queues+0x101/0x280
ata_piix
hrtimer_interrupt+0x100/0x220
libata
smp_apic_timer_interrupt+0x6a/0x130
crc32c_intel
apic_timer_interrupt+0xf/0x20
serio_raw
</IRQ>
virtio_net
RIP: 0010:native_safe_halt+0xe/0x20
net_failover
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-135vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-135vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-135vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-135vm1.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-135vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-135vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-135vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-135vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.25.61@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.25.61@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Return to new crashes list