Editing crashreport #73999

ReasonCrashing FunctionWhere to cut BacktraceReports Count
watchdog: BUG: soft lockup - native_safe_haltkvm_wait
__pv_queued_spin_lock_slowpath
_raw_spin_lock
lnet_discover_peer_locked
lnet_discover_peer_nid
LNetAddPeer
class_add_uuid
lustre_lwp_setup
client_lwp_config_process
llog_process_thread
llog_process_thread_daemonize
kthread
ret_from_fork
panic
watchdog_timer_fn
__hrtimer_run_queues
hrtimer_interrupt
smp_apic_timer_interrupt
1

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [llog_process_th:13065]
Modules linked in: mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev virtio_balloon pcspkr i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix crc32c_intel libata virtio_net virtio_blk serio_raw net_failover failover
watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [lnet_discovery:12697]
CPU: 0 PID: 13065 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
Modules linked in:
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
mdd(OE) lod(OE)
RIP: 0010:native_safe_halt+0xe/0x20
mdt(OE)
Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 42 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
lfsck(OE)
RSP: 0018:ffffac154109fb38 EFLAGS: 00000246
mgs(OE) mgc(OE)
ORIG_RAX: ffffffffffffff13
osd_ldiskfs(OE)
RAX: 0000000000000001 RBX: ffff8a7843c716b4 RCX: 0000000000000001
lquota(OE)
RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff8a78ffc345d4
lustre(OE)
RBP: ffff8a78ffc345c0 R08: 0000000000000004 R09: 0000000000000000
mdc(OE)
R10: ffffac154109fcb0 R11: 0000000000000049 R12: ffff8a78ffd345c0
lov(OE)
R13: ffff8a78ffc345d4 R14: 0000000000000001 R15: 0000000000040000
osc(OE)
FS: 0000000000000000(0000) GS:ffff8a78ffc00000(0000) knlGS:0000000000000000
lmv(OE)
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
fid(OE)
CR2: 000055a3af34af78 CR3: 000000005b410003 CR4: 00000000000606f0
fld(OE)
Call Trace:
ksocklnd(OE)
<IRQ>
ptlrpc(OE)
? watchdog_timer_fn.cold.10+0x46/0x9e
obdclass(OE)
? watchdog+0x30/0x30
lnet(OE)
? __hrtimer_run_queues+0x101/0x280
ldiskfs(OE)
? hrtimer_interrupt+0x100/0x220
libcfs(OE)
? smp_apic_timer_interrupt+0x6a/0x130
dm_flakey
? apic_timer_interrupt+0xf/0x20
dm_mod
</IRQ>
rpcsec_gss_krb5
? native_safe_halt+0xe/0x20
auth_rpcgss
kvm_wait+0x58/0x60
nfsv4 dns_resolver
__pv_queued_spin_lock_slowpath+0x219/0x2a0
nfs
_raw_spin_lock+0x1e/0x30
lockd
lnet_discover_peer_locked+0x99/0x460 [lnet]
grace
? finish_wait+0x80/0x80
fscache
lnet_discover_peer_nid+0x47/0x90 [lnet]
intel_rapl_msr
LNetAddPeer+0x52a/0x7c0 [lnet]
intel_rapl_common
? free_one_page+0x3c1/0x530
crct10dif_pclmul
class_add_uuid+0x2ed/0x520 [obdclass]
crc32_pclmul
lustre_lwp_setup+0xe7/0xa80 [ptlrpc]
ghash_clmulni_intel
client_lwp_config_process+0x6e5/0x890 [ptlrpc]
joydev
? llog_client_next_block+0x29a/0x460 [ptlrpc]
virtio_balloon
llog_process_thread+0xb21/0x1a30 [obdclass]
pcspkr
? llog_validate+0x370/0x370 [obdclass]
i2c_piix4
llog_process_thread_daemonize+0x70/0x90 [obdclass]
sunrpc
kthread+0x134/0x150
ext4
? set_kthread_struct+0x50/0x50
mbcache
ret_from_fork+0x35/0x40
jbd2
Kernel panic - not syncing: softlockup: hung tasks
ata_generic
CPU: 0 PID: 13065 Comm: llog_process_th Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
ata_piix
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
crc32c_intel
Call Trace:
libata
<IRQ>
virtio_net
dump_stack+0x41/0x60
virtio_blk
panic+0xe7/0x2ac
serio_raw
? syscall_return_via_sysret+0x6e/0x94
net_failover
watchdog_timer_fn.cold.10+0x85/0x9e
failover
? watchdog+0x30/0x30
CPU: 1 PID: 12697 Comm: lnet_discovery Kdump: loaded Tainted: G OEL -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1
__hrtimer_run_queues+0x101/0x280
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
hrtimer_interrupt+0x100/0x220
RIP: 0010:native_safe_halt+0xe/0x20
smp_apic_timer_interrupt+0x6a/0x130
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm3.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm7.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-81vm6.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt1
LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_MDS/mdt3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.201@tcp --fsname=lustre --mdt --index=2 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=2034237 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_MDS/mdt3
LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_MDS/mdt1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_MDS/mdt1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create mds1_flakey --table "0 4071424 linear /dev/vg_Role_MDS/mdt1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.58_105_g6d1c598
LNet: Added LNI 10.240.42.201@tcp [8/256/0/180]
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/usr/sbin/l_getidentity in log lustre-MDT0000
Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
Lustre: lustre-MDT0000: new disk, initializing
Link to test
Return to new crashes list