Editing crashreport #74636

ReasonCrashing FunctionWhere to cut BacktraceReports Count
kernel BUG at __list_add_valid__mutex_add_waiter
__mutex_lock
ll_merge_attr
cl_glimpse_lock
__cl_glimpse_size
ll_getattr_dentry
svcxdr_encode_post_op_attr
nfs3svc_encode_readres
nfsd_dispatch
svc_process_common
svc_process
svc_handle_xprt
svc_recv
nfsd
kthread
ret_from_fork
2

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
parallel-scale-nfsv3 test iorssf: iorssf
kernel BUG at lib/list_debug.c:26!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 PID: 68162 Comm: nfsd Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:__list_add_valid.cold+0x3d/0x3f
Code: f2 4c 89 c1 48 89 fe 48 c7 c7 f8 bf 67 98 e8 5d 77 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 a0 bf 67 98 e8 46 77 fe ff <0f> 0b 48 89 fe 48 c7 c7 30 c0 67 98 e8 35 77 fe ff 0f 0b 48 c7 c7
RSP: 0018:ff5edbacc783baf0 EFLAGS: 00010246
RAX: 0000000000000075 RBX: ff5edbacc783bb30 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff3c85333bb208c0 RDI: ff3c85333bb208c0
RBP: ff3c8531f2b3fb90 R08: 0000000000000000 R09: ff5edbacc783b9b0
R10: ff5edbacc783b9a8 R11: ffffffff991e93e8 R12: ff3c8531f2b3fb80
R13: ff3c8531f2b3fb90 R14: ff5edbacc783bb30 R15: ff3c8531f2b3fb88
FS: 0000000000000000(0000) GS:ff3c85333bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000556ef0d5b3c8 CR3: 0000000102d66001 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x26e/0x2df
? show_trace_log_lvl+0x26e/0x2df
? __mutex_add_waiter+0x23/0x60
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xce/0x120
? __list_add_valid.cold+0x3d/0x3f
? do_error_trap+0x65/0x80
? __list_add_valid.cold+0x3d/0x3f
? exc_invalid_op+0x4e/0x70
? __list_add_valid.cold+0x3d/0x3f
? asm_exc_invalid_op+0x16/0x20
? __list_add_valid.cold+0x3d/0x3f
__mutex_add_waiter+0x23/0x60
__mutex_lock.constprop.0+0x2a6/0x6a0
? __pfx_autoremove_wake_function+0x10/0x10
ll_merge_attr+0x16/0x40 [lustre]
cl_glimpse_lock+0x241/0x2b0 [lustre]
__cl_glimpse_size+0x166/0x2a0 [lustre]
ll_getattr_dentry+0xad4/0xcb0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
svcxdr_encode_post_op_attr+0x89/0x130 [nfsd]
nfs3svc_encode_readres+0x4d/0x130 [nfsd]
nfsd_dispatch+0x108/0x220 [nfsd]
svc_process_common+0x2e4/0x650 [sunrpc]
? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
svc_process+0x12d/0x170 [sunrpc]
svc_handle_xprt+0x448/0x580 [sunrpc]
svc_recv+0x17a/0x2c0 [sunrpc]
? __pfx_nfsd+0x10/0x10 [nfsd]
nfsd+0x84/0xb0 [nfsd]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core nfsd nfs_acl tls dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support i2c_i801 lpc_ich pcspkr i2c_smbus virtio_balloon joydev drm fuse ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net net_failover failover virtio_blk serio_raw [last unloaded: dm_flakey]
list_add corruption. prev->next should be next (ff3c8531f2b3fb90), but was 00000000f2b3fb90. (prev=ff3c8531f2b3fb90).
------------[ cut here ]------------
Link to test
parallel-scale-nfsv3 test connectathon: connectathon
kernel BUG at lib/list_debug.c:26!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 983082 Comm: nfsd Kdump: loaded Tainted: G W OE ------ --- 5.14.0-611.13.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:__list_add_valid.cold+0x3d/0x3f
Code: f2 4c 89 c1 48 89 fe 48 c7 c7 08 29 32 85 e8 2a 61 fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 b0 28 32 85 e8 13 61 fe ff <0f> 0b 48 89 fe 48 c7 c7 40 29 32 85 e8 02 61 fe ff 0f 0b 48 c7 c7
RSP: 0018:ff733b2880a43af0 EFLAGS: 00010246
RAX: 0000000000000075 RBX: ff733b2880a43b30 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff2d90d6ffa20900 RDI: ff2d90d6ffa20900
RBP: ff2d90d6b6f08998 R08: 0000000000000000 R09: ff733b2880a439b0
R10: ff733b2880a439a8 R11: ffffffff85de2348 R12: ff2d90d6b6f08988
R13: ff2d90d6b6f08998 R14: ff733b2880a43b30 R15: ff2d90d6b6f08990
FS: 0000000000000000(0000) GS:ff2d90d6ffa00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9a4b7cb4e0 CR3: 0000000100900004 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x26e/0x2df
? show_trace_log_lvl+0x26e/0x2df
? __mutex_add_waiter+0x23/0x60
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xcd/0x120
? __list_add_valid.cold+0x3d/0x3f
? do_error_trap+0x65/0x80
? __list_add_valid.cold+0x3d/0x3f
? exc_invalid_op+0x4e/0x70
? __list_add_valid.cold+0x3d/0x3f
? asm_exc_invalid_op+0x16/0x20
? __list_add_valid.cold+0x3d/0x3f
__mutex_add_waiter+0x23/0x60
__mutex_lock.constprop.0+0x29f/0x700
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
ll_merge_attr+0x16/0x40 [lustre]
cl_glimpse_lock+0x241/0x2b0 [lustre]
__cl_glimpse_size+0x166/0x2a0 [lustre]
ll_getattr_dentry+0xad2/0xcb0 [lustre]
svcxdr_encode_post_op_attr+0x89/0x130 [nfsd]
nfs3svc_encode_readres+0x4d/0x130 [nfsd]
nfsd_dispatch+0x108/0x220 [nfsd]
svc_process_common+0x2ee/0x660 [sunrpc]
? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
svc_process+0x12d/0x180 [sunrpc]
svc_handle_xprt+0x448/0x580 [sunrpc]
? schedule+0x2c/0xb0
svc_recv+0xe0/0x200 [sunrpc]
nfsd+0x83/0xd0 [nfsd]
? __pfx_nfsd+0x10/0x10 [nfsd]
kthread+0x101/0x110
? __pfx_kthread+0x10/0x10
ret_from_fork+0x28/0x50
</TASK>
Modules linked in: rpcrdma rdma_cm iw_cm ib_cm ib_core nfsd nfs_acl dm_flakey tls obdecho(OE) ec(OE) ptlrpc_gss(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support i2c_i801 virtio_balloon lpc_ich pcspkr i2c_smbus joydev fuse dm_mod drm ext4 mbcache jbd2 crct10dif_pclmul ahci libahci crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_blk virtio_net net_failover failover serio_raw [last unloaded: dm_flakey]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark bash .\/runtests -N 10 -b -f \/mnt\/lustre\/d0.parallel-scale-nfs\/d0.connectathon
Lustre: DEBUG MARKER: bash ./runtests -N 10 -b -f /mnt/lustre/d0.parallel-scale-nfs/d0.connectathon
Lustre: DEBUG MARKER: /usr/sbin/lctl mark bash .\/runtests -N 10 -g -f \/mnt\/lustre\/d0.parallel-scale-nfs\/d0.connectathon
Lustre: DEBUG MARKER: bash ./runtests -N 10 -g -f /mnt/lustre/d0.parallel-scale-nfs/d0.connectathon
Lustre: DEBUG MARKER: /usr/sbin/lctl mark bash .\/runtests -N 10 -s -f \/mnt\/lustre\/d0.parallel-scale-nfs\/d0.connectathon
Lustre: DEBUG MARKER: bash ./runtests -N 10 -s -f /mnt/lustre/d0.parallel-scale-nfs/d0.connectathon
list_add corruption. prev->next should be next (ff2d90d6b6f08998), but was 00000000b6f08998. (prev=ff2d90d6b6f08998).
------------[ cut here ]------------
Link to test
Return to new crashes list