Editing crashreport #73998

ReasonCrashing FunctionWhere to cut BacktraceReports Count
watchdog: BUG: soft lockup - native_safe_haltkvm_wait
kvm_wait
__pv_queued_spin_lock_slowpath
__pv_queued_spin_lock_slowpath
_raw_spin_lock
_raw_spin_lock
lnet_discover_peer_locked
lnet_peer_discovery
lnet_discover_peer_nid
LNetAddPeer
kthread
class_add_uuid
lustre_lwp_setup
ret_from_fork
client_lwp_config_process
llog_process_thread
llog_process_thread_daemonize
kthread
panic
ret_from_fork
watchdog_timer_fn
__hrtimer_run_queues
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
1

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [llog_process_th:13044]
watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [lnet_discovery:12676]
Modules linked in:
Modules linked in:
mdd(OE)
mdd(OE)
lod(OE)
lod(OE)
mdt(OE)
mdt(OE)
lfsck(OE)
lfsck(OE)
mgs(OE)
mgs(OE)
mgc(OE)
mgc(OE)
osd_ldiskfs(OE)
osd_ldiskfs(OE)
lquota(OE)
lquota(OE)
lustre(OE)
lustre(OE)
mdc(OE)
mdc(OE)
lov(OE)
lov(OE)
osc(OE)
osc(OE)
lmv(OE)
lmv(OE)
fid(OE)
fid(OE)
fld(OE)
fld(OE)
ksocklnd(OE)
ksocklnd(OE)
ptlrpc(OE)
ptlrpc(OE)
obdclass(OE)
obdclass(OE)
lnet(OE)
lnet(OE)
ldiskfs(OE)
ldiskfs(OE)
libcfs(OE)
libcfs(OE)
dm_flakey
dm_flakey
dm_mod
dm_mod
rpcsec_gss_krb5
rpcsec_gss_krb5
auth_rpcgss
auth_rpcgss
nfsv4
nfsv4
dns_resolver
dns_resolver
nfs
nfs
lockd
lockd
grace
grace
fscache
fscache
intel_rapl_msr
intel_rapl_msr
intel_rapl_common
intel_rapl_common
crct10dif_pclmul
crct10dif_pclmul
crc32_pclmul
crc32_pclmul
ghash_clmulni_intel
ghash_clmulni_intel
joydev
joydev
pcspkr
pcspkr
virtio_balloon
virtio_balloon
i2c_piix4
i2c_piix4
sunrpc
sunrpc
ext4
ext4
mbcache
mbcache
jbd2
jbd2
ata_generic
ata_generic
ata_piix
ata_piix
libata
libata
crc32c_intel
crc32c_intel
virtio_net
virtio_net
serio_raw
serio_raw
net_failover
net_failover
failover
failover
virtio_blk
virtio_blk
CPU: 0 PID: 13044 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
CPU: 1 PID: 12676 Comm: lnet_discovery Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:native_safe_halt+0xe/0x20
RIP: 0010:native_safe_halt+0xe/0x20
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
RSP: 0018:ffffa019c107fb38 EFLAGS: 00000246
RSP: 0018:ffffa019c0e87e00 EFLAGS: 00000246
ORIG_RAX: ffffffffffffff13
ORIG_RAX: ffffffffffffff13
RAX: 0000000000000001 RBX: ffff92626424a2b4 RCX: 0000000000000001
RAX: 0000000000000003 RBX: ffff92626424a2b4 RCX: 0000000000000008
RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff92627bc345d4
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff92626424a2b4
RBP: ffff92627bc345c0 R08: 0000000000000004 R09: 0000000000000000
RBP: ffff92627bd345c0 R08: 0000000000000008 R09: 0000000000000058
R10: ffffa019c107fcb0 R11: 0000000000000049 R12: ffff92627bd345c0
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff92627bc345d4 R14: 0000000000000001 R15: 0000000000040000
R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000080000
FS: 0000000000000000(0000) GS:ffff92627bc00000(0000) knlGS:0000000000000000
FS: 0000000000000000(0000) GS:ffff92627bd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056203de8b3a7 CR3: 0000000121810001 CR4: 00000000003706f0
CR2: 000056203dea25a8 CR3: 0000000121810002 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
Call Trace:
<IRQ>
<IRQ>
? watchdog_timer_fn.cold.10+0x46/0x9e
? watchdog_timer_fn.cold.10+0x46/0x9e
? watchdog+0x30/0x30
? watchdog+0x30/0x30
? __hrtimer_run_queues+0x101/0x280
? __hrtimer_run_queues+0x101/0x280
? hrtimer_interrupt+0x100/0x220
? hrtimer_interrupt+0x100/0x220
? smp_apic_timer_interrupt+0x6a/0x130
? smp_apic_timer_interrupt+0x6a/0x130
? apic_timer_interrupt+0xf/0x20
? apic_timer_interrupt+0xf/0x20
</IRQ>
</IRQ>
? native_safe_halt+0xe/0x20
? native_safe_halt+0xe/0x20
kvm_wait+0x58/0x60
kvm_wait+0x58/0x60
__pv_queued_spin_lock_slowpath+0x219/0x2a0
__pv_queued_spin_lock_slowpath+0x268/0x2a0
_raw_spin_lock+0x1e/0x30
_raw_spin_lock+0x1e/0x30
lnet_discover_peer_locked+0x99/0x460 [lnet]
lnet_peer_discovery+0xccc/0x1b90 [lnet]
? finish_wait+0x80/0x80
? finish_task_switch+0x86/0x2f0
lnet_discover_peer_nid+0x47/0x90 [lnet]
? finish_wait+0x80/0x80
LNetAddPeer+0x52a/0x7c0 [lnet]
? lnet_peer_merge_data+0x10c0/0x10c0 [lnet]
? free_one_page+0x3c1/0x530
kthread+0x134/0x150
class_add_uuid+0x2ed/0x520 [obdclass]
? set_kthread_struct+0x50/0x50
lustre_lwp_setup+0xe7/0xa80 [ptlrpc]
ret_from_fork+0x35/0x40
client_lwp_config_process+0x6e5/0x890 [ptlrpc]
Kernel panic - not syncing: softlockup: hung tasks
? llog_client_next_block+0x29a/0x460 [ptlrpc]
CPU: 1 PID: 12676 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
llog_process_thread+0xb21/0x1a30 [obdclass]
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
? llog_validate+0x370/0x370 [obdclass]
Call Trace:
llog_process_thread_daemonize+0x70/0x90 [obdclass]
<IRQ>
kthread+0x134/0x150
dump_stack+0x41/0x60
? set_kthread_struct+0x50/0x50
panic+0xe7/0x2ac
ret_from_fork+0x35/0x40
? syscall_return_via_sysret+0x6e/0x94
watchdog_timer_fn.cold.10+0x85/0x9e
? watchdog+0x30/0x30
__hrtimer_run_queues+0x101/0x280
hrtimer_interrupt+0x100/0x220
smp_apic_timer_interrupt+0x6a/0x130
apic_timer_interrupt+0xf/0x20
</IRQ>
RIP: 0010:native_safe_halt+0xe/0x20
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-134vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-134vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-134vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-134vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-134vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-134vm10.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-134vm3.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-134vm11.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.177@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.24.177@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Return to new crashes list