Editing crashreport #69750

ReasonCrashing FunctionWhere to cut BacktraceReports Count
BUG: kernel NULL pointer dereferencedbuf_writedbuf_sync_leaf
dbuf_sync_list
dbuf_sync_indirect
dbuf_sync_list
dbuf_sync_indirect
dbuf_sync_list
dnode_sync
sync_dnodes_task
taskq_thread
kthread
ret_from_fork
3

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity test 136: Race catalog processing 2
BUG: kernel NULL pointer dereference, address: 0000000000000018
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 888386 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:dbuf_write+0x18e/0x540 [zfs]
Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50
RSP: 0018:ffff9c4bc22bba48 EFLAGS: 00010202
RAX: 0000000000000004 RBX: ffff894fdd658000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff894ff81ca550 RDI: ffff894ffe848400
RBP: ffff894fd4b1c780 R08: ffff89503fc2b638 R09: ffff89503fc2b638
R10: 0000000000000355 R11: 0000000000000002 R12: ffff89500ac3e2e0
R13: ffff894fd4b1c848 R14: ffff89500ac3e2e0 R15: ffff894ffe848400
FS: 0000000000000000(0000) GS:ffff89503fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000018 CR3: 0000000034820004 CR4: 00000000000606f0
Call Trace:
<TASK>
? finish_task_switch.isra.0+0x8c/0x2a0
? __schedule+0x250/0x620
dbuf_sync_leaf+0x13e/0x550 [zfs]
? zio_gang_tree_free+0x70/0x70 [zfs]
? taskq_dispatch_ent+0x107/0x1c0 [spl]
? _raw_spin_unlock_irqrestore+0xa/0x30
? zio_issue_async+0x4f/0x90 [zfs]
? zio_nowait+0xb4/0x1a0 [zfs]
dbuf_sync_list+0xc1/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dnode_sync+0x3e3/0x750 [zfs]
? dnode_multilist_index_func+0x94/0xb0 [zfs]
sync_dnodes_task+0x71/0xa0 [zfs]
taskq_thread+0x2cf/0x500 [spl]
? wake_up_q+0x90/0x90
? taskq_thread_spawn+0x60/0x60 [spl]
kthread+0xd9/0x100
? kthread_complete_and_exit+0x20/0x20
ret_from_fork+0x22/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul crc32_pclmul libata crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw [last unloaded: llog_test]
CR2: 0000000000000018
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Autotest: Test running for 210 minutes (lustre-master_rolling-upgrade-client1-zfs_4522.164)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout"
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Link to test
sanity test 136: Race catalog processing 2
BUG: kernel NULL pointer dereference, address: 0000000000000018
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 1 PID: 887784 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:dbuf_write+0x18e/0x540 [zfs]
Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50
RSP: 0018:ffffb7ef00647a48 EFLAGS: 00010206
RAX: 0000000000000004 RBX: ffff97d4e8c62d80 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff97d4f6320d20 RDI: ffff97d4fe503800
RBP: ffff97d4f0052d80 R08: 0000000000000000 R09: ffffb7ef00647af0
R10: ffffb7ef00647a00 R11: 0000000000000007 R12: ffff97d482daaaa0
R13: ffff97d4f0052e48 R14: ffff97d482daaaa0 R15: ffff97d4fe503800
FS: 0000000000000000(0000) GS:ffff97d53fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000018 CR3: 0000000036286002 CR4: 00000000001706e0
Call Trace:
<TASK>
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up_common_lock+0x8a/0xc0
dbuf_sync_leaf+0x13e/0x550 [zfs]
? taskq_dispatch_ent+0x141/0x1c0 [spl]
? _raw_spin_unlock_irqrestore+0xa/0x30
? zio_issue_async+0x4f/0x90 [zfs]
? zio_nowait+0xb4/0x1a0 [zfs]
dbuf_sync_list+0xc1/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dnode_sync+0x3e3/0x750 [zfs]
? dnode_multilist_index_func+0x94/0xb0 [zfs]
sync_dnodes_task+0x71/0xa0 [zfs]
taskq_thread+0x2cf/0x500 [spl]
? wake_up_q+0x90/0x90
? taskq_thread_spawn+0x60/0x60 [spl]
kthread+0xd9/0x100
? kthread_complete_and_exit+0x20/0x20
ret_from_fork+0x22/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr i2c_piix4 virtio_balloon joydev drm fuse dm_mod ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul crc32_pclmul crc32c_intel libata virtio_net net_failover ghash_clmulni_intel failover virtio_blk serio_raw [last unloaded: llog_test]
CR2: 0000000000000018
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Link to test
sanity test 136: Race catalog processing 2
BUG: kernel NULL pointer dereference, address: 0000000000000018
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 759669 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:dbuf_write+0x18e/0x540 [zfs]
Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50
RSP: 0018:fffface181a73a48 EFLAGS: 00010287
RAX: 0000000000000004 RBX: ffff901175735b00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff901157e970a0 RDI: ffff90115d39b600
RBP: ffff901192902780 R08: 0000000000000000 R09: fffface181a73af0
R10: fffface181a73a00 R11: 0000000000000007 R12: ffff90116b6c2aa0
R13: ffff901192902848 R14: ffff90116b6c2aa0 R15: ffff90115d39b600
FS: 0000000000000000(0000) GS:ffff9011bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000018 CR3: 0000000034c8a005 CR4: 00000000000606f0
Call Trace:
<TASK>
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up_common_lock+0x8a/0xc0
dbuf_sync_leaf+0x13e/0x550 [zfs]
? taskq_dispatch_ent+0x141/0x1c0 [spl]
? _raw_spin_unlock_irqrestore+0xa/0x30
? zio_issue_async+0x4f/0x90 [zfs]
? zio_nowait+0xb4/0x1a0 [zfs]
dbuf_sync_list+0xc1/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dbuf_sync_indirect+0xe3/0x170 [zfs]
dbuf_sync_list+0x50/0x120 [zfs]
dnode_sync+0x3e3/0x750 [zfs]
? dnode_multilist_index_func+0x94/0xb0 [zfs]
sync_dnodes_task+0x71/0xa0 [zfs]
taskq_thread+0x2cf/0x500 [spl]
? wake_up_q+0x90/0x90
? taskq_thread_spawn+0x60/0x60 [spl]
kthread+0xd9/0x100
? kthread_complete_and_exit+0x20/0x20
ret_from_fork+0x22/0x30
</TASK>
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common virtio_balloon pcspkr i2c_piix4 joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ata_piix ghash_clmulni_intel libata virtio_blk net_failover failover serio_raw [last unloaded: llog_test]
CR2: 0000000000000018
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Link to test
Return to new crashes list