Editing crashreport #73997

ReasonCrashing FunctionWhere to cut BacktraceReports Count
watchdog: BUG: soft lockup - native_safe_haltkvm_wait
__pv_queued_spin_lock_slowpath
_raw_spin_lock
lnet_discover_peer_locked
lnet_discover_peer_nid
LNetAddPeer
class_add_uuid
lustre_lwp_setup
client_lwp_config_process
llog_process_thread
llog_process_thread_daemonize
kthread
ret_from_fork
panic
watchdog_timer_fn
__hrtimer_run_queues
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
2

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [llog_process_th:13049]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix virtio_net crc32c_intel libata serio_raw virtio_blk net_failover failover
watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [lnet_discovery:12681]
CPU: 0 PID: 13049 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Modules linked in:
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
mdd(OE) lod(OE)
RIP: 0010:native_safe_halt+0xe/0x20
mdt(OE)
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
lfsck(OE)
RSP: 0018:ffffba5e410d3b38 EFLAGS: 00000246
mgs(OE)
ORIG_RAX: ffffffffffffff13
mgc(OE)
RAX: 0000000000000001 RBX: ffff950b0617a6b4 RCX: 0000000000000001
osd_ldiskfs(OE)
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff950bbfc345d4
lquota(OE)
RBP: ffff950bbfc345c0 R08: 0000000000000004 R09: 0000000000000000
lustre(OE)
R10: ffffba5e410d3cb0 R11: 0000000000000049 R12: ffff950bbfd345c0
mdc(OE)
R13: ffff950bbfc345d4 R14: 0000000000000001 R15: 0000000000040000
lov(OE)
FS: 0000000000000000(0000) GS:ffff950bbfc00000(0000) knlGS:0000000000000000
osc(OE)
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
lmv(OE)
CR2: 00005604d406d7b4 CR3: 0000000076a10001 CR4: 00000000000606f0
fid(OE)
Call Trace:
fld(OE)
<IRQ>
ksocklnd(OE)
? watchdog_timer_fn.cold.10+0x46/0x9e
ptlrpc(OE)
? watchdog+0x30/0x30
obdclass(OE)
? __hrtimer_run_queues+0x101/0x280
lnet(OE)
? hrtimer_interrupt+0x100/0x220
ldiskfs(OE)
? smp_apic_timer_interrupt+0x6a/0x130
libcfs(OE)
? apic_timer_interrupt+0xf/0x20
dm_flakey
</IRQ>
dm_mod
? native_safe_halt+0xe/0x20
rpcsec_gss_krb5
kvm_wait+0x58/0x60
auth_rpcgss
__pv_queued_spin_lock_slowpath+0x219/0x2a0
nfsv4
_raw_spin_lock+0x1e/0x30
dns_resolver
lnet_discover_peer_locked+0x99/0x460 [lnet]
nfs
? finish_wait+0x80/0x80
lockd
lnet_discover_peer_nid+0x47/0x90 [lnet]
grace
LNetAddPeer+0x52a/0x7c0 [lnet]
fscache
? free_one_page+0x3c1/0x530
intel_rapl_msr
class_add_uuid+0x2ed/0x520 [obdclass]
intel_rapl_common
lustre_lwp_setup+0xe7/0xa80 [ptlrpc]
crct10dif_pclmul
client_lwp_config_process+0x6e5/0x890 [ptlrpc]
crc32_pclmul
? llog_client_next_block+0x29a/0x460 [ptlrpc]
ghash_clmulni_intel
llog_process_thread+0xb21/0x1a30 [obdclass]
joydev
? llog_validate+0x370/0x370 [obdclass]
pcspkr
llog_process_thread_daemonize+0x70/0x90 [obdclass]
virtio_balloon
kthread+0x134/0x150
i2c_piix4
? set_kthread_struct+0x50/0x50
sunrpc
ret_from_fork+0x35/0x40
ext4
Kernel panic - not syncing: softlockup: hung tasks
mbcache
CPU: 0 PID: 13049 Comm: llog_process_th Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
jbd2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
ata_generic
Call Trace:
ata_piix
<IRQ>
virtio_net
dump_stack+0x41/0x60
crc32c_intel
panic+0xe7/0x2ac
libata
? syscall_return_via_sysret+0x6e/0x94
serio_raw
watchdog_timer_fn.cold.10+0x85/0x9e
virtio_blk
? watchdog+0x30/0x30
net_failover
__hrtimer_run_queues+0x101/0x280
failover
hrtimer_interrupt+0x100/0x220
smp_apic_timer_interrupt+0x6a/0x130
CPU: 1 PID: 12681 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
apic_timer_interrupt+0xf/0x20
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
</IRQ>
RIP: 0010:native_safe_halt+0xe/0x20
RIP: 0010:native_safe_halt+0xe/0x20
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-25vm8.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.38.106@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.38.106@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [llog_process_th:13038]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs intel_rapl_msr lockd grace fscache intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon pcspkr joydev i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net serio_raw virtio_blk net_failover failover
watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [lnet_discovery:12670]
CPU: 0 PID: 13038 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Modules linked in:
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
mdd(OE)
RIP: 0010:native_safe_halt+0xe/0x20
lod(OE)
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
mdt(OE)
RSP: 0018:ffff991b0105fb38 EFLAGS: 00000246
lfsck(OE)
ORIG_RAX: ffffffffffffff13
mgs(OE)
RAX: 0000000000000001 RBX: ffff8cb6c3d9a8b4 RCX: 0000000000000001
mgc(OE)
RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff8cb6fbc345d4
osd_ldiskfs(OE)
RBP: ffff8cb6fbc345c0 R08: 0000000000000004 R09: 0000000000000000
lquota(OE)
R10: ffff991b0105fcb0 R11: 0000000000000049 R12: ffff8cb6fbd345c0
lustre(OE)
R13: ffff8cb6fbc345d4 R14: 0000000000000001 R15: 0000000000040000
mdc(OE)
FS: 0000000000000000(0000) GS:ffff8cb6fbc00000(0000) knlGS:0000000000000000
lov(OE)
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
osc(OE)
CR2: 00005563f398faf0 CR3: 00000000a8610003 CR4: 00000000003706f0
lmv(OE)
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
fid(OE)
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
fld(OE)
Call Trace:
ksocklnd(OE)
<IRQ>
ptlrpc(OE)
? watchdog_timer_fn.cold.10+0x46/0x9e
obdclass(OE)
? watchdog+0x30/0x30
lnet(OE)
? __hrtimer_run_queues+0x101/0x280
ldiskfs(OE)
? hrtimer_interrupt+0x100/0x220
libcfs(OE)
? smp_apic_timer_interrupt+0x6a/0x130
dm_flakey
? apic_timer_interrupt+0xf/0x20
dm_mod
</IRQ>
rpcsec_gss_krb5
? native_safe_halt+0xe/0x20
auth_rpcgss
kvm_wait+0x58/0x60
nfsv4
__pv_queued_spin_lock_slowpath+0x219/0x2a0
dns_resolver
_raw_spin_lock+0x1e/0x30
nfs
lnet_discover_peer_locked+0x99/0x460 [lnet]
intel_rapl_msr
? finish_wait+0x80/0x80
lockd
lnet_discover_peer_nid+0x47/0x90 [lnet]
grace
LNetAddPeer+0x52a/0x7c0 [lnet]
fscache
? free_one_page+0x3c1/0x530
intel_rapl_common
class_add_uuid+0x2ed/0x520 [obdclass]
crct10dif_pclmul
lustre_lwp_setup+0xe7/0xa80 [ptlrpc]
crc32_pclmul
client_lwp_config_process+0x6e5/0x890 [ptlrpc]
ghash_clmulni_intel
? llog_client_next_block+0x29a/0x460 [ptlrpc]
virtio_balloon
llog_process_thread+0xb21/0x1a30 [obdclass]
pcspkr
? llog_validate+0x370/0x370 [obdclass]
joydev i2c_piix4
llog_process_thread_daemonize+0x70/0x90 [obdclass]
sunrpc
kthread+0x134/0x150
ext4
? set_kthread_struct+0x50/0x50
mbcache
ret_from_fork+0x35/0x40
jbd2
Kernel panic - not syncing: softlockup: hung tasks
ata_generic
CPU: 0 PID: 13038 Comm: llog_process_th Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
ata_piix
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
libata
Call Trace:
crc32c_intel
<IRQ>
virtio_net
dump_stack+0x41/0x60
serio_raw
panic+0xe7/0x2ac
virtio_blk
? syscall_return_via_sysret+0x6e/0x94
net_failover
watchdog_timer_fn.cold.10+0x85/0x9e
failover
? watchdog+0x30/0x30
__hrtimer_run_queues+0x101/0x280
CPU: 1 PID: 12670 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
hrtimer_interrupt+0x100/0x220
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
smp_apic_timer_interrupt+0x6a/0x130
RIP: 0010:native_safe_halt+0xe/0x20
apic_timer_interrupt+0xf/0x20
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-137vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-137vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-137vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-137vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-137vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-137vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-137vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-137vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.22.169@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.22.169@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Return to new crashes list