Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
Limit to a test: (Copy from below "Failing text"): | |
Delete these reports as invalid (real bug in review or some such) | |
Bug or comment: | |
Extra info: |
Failing Test | Full Crash | Messages before crash | Comment |
---|---|---|---|
sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000018 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 888386 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:dbuf_write+0x18e/0x540 [zfs] Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50 RSP: 0018:ffff9c4bc22bba48 EFLAGS: 00010202 RAX: 0000000000000004 RBX: ffff894fdd658000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff894ff81ca550 RDI: ffff894ffe848400 RBP: ffff894fd4b1c780 R08: ffff89503fc2b638 R09: ffff89503fc2b638 R10: 0000000000000355 R11: 0000000000000002 R12: ffff89500ac3e2e0 R13: ffff894fd4b1c848 R14: ffff89500ac3e2e0 R15: ffff894ffe848400 FS: 0000000000000000(0000) GS:ffff89503fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000018 CR3: 0000000034820004 CR4: 00000000000606f0 Call Trace: <TASK> ? finish_task_switch.isra.0+0x8c/0x2a0 ? __schedule+0x250/0x620 dbuf_sync_leaf+0x13e/0x550 [zfs] ? zio_gang_tree_free+0x70/0x70 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul crc32_pclmul libata crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw [last unloaded: llog_test] CR2: 0000000000000018 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Autotest: Test running for 210 minutes (lustre-master_rolling-upgrade-client1-zfs_4522.164) Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout" Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000018 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 1 PID: 887784 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:dbuf_write+0x18e/0x540 [zfs] Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50 RSP: 0018:ffffb7ef00647a48 EFLAGS: 00010206 RAX: 0000000000000004 RBX: ffff97d4e8c62d80 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff97d4f6320d20 RDI: ffff97d4fe503800 RBP: ffff97d4f0052d80 R08: 0000000000000000 R09: ffffb7ef00647af0 R10: ffffb7ef00647a00 R11: 0000000000000007 R12: ffff97d482daaaa0 R13: ffff97d4f0052e48 R14: ffff97d482daaaa0 R15: ffff97d4fe503800 FS: 0000000000000000(0000) GS:ffff97d53fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000018 CR3: 0000000036286002 CR4: 00000000001706e0 Call Trace: <TASK> ? _raw_spin_unlock_irqrestore+0xa/0x30 ? __wake_up_common_lock+0x8a/0xc0 dbuf_sync_leaf+0x13e/0x550 [zfs] ? taskq_dispatch_ent+0x141/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr i2c_piix4 virtio_balloon joydev drm fuse dm_mod ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul crc32_pclmul crc32c_intel libata virtio_net net_failover ghash_clmulni_intel failover virtio_blk serio_raw [last unloaded: llog_test] CR2: 0000000000000018 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000018 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 759669 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:dbuf_write+0x18e/0x540 [zfs] Code: 28 4d 89 fc 4c 8b 7c 24 30 0f b6 55 68 8b 85 d8 00 00 00 48 8b 5c 24 10 49 39 5c 24 50 0f 84 36 02 00 00 48 8b 8b 30 01 00 00 <4c> 8b 69 18 48 8b 3c 24 31 c9 48 8b 37 48 85 f6 74 07 48 8b 8e 50 RSP: 0018:fffface181a73a48 EFLAGS: 00010287 RAX: 0000000000000004 RBX: ffff901175735b00 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff901157e970a0 RDI: ffff90115d39b600 RBP: ffff901192902780 R08: 0000000000000000 R09: fffface181a73af0 R10: fffface181a73a00 R11: 0000000000000007 R12: ffff90116b6c2aa0 R13: ffff901192902848 R14: ffff90116b6c2aa0 R15: ffff90115d39b600 FS: 0000000000000000(0000) GS:ffff9011bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000018 CR3: 0000000034c8a005 CR4: 00000000000606f0 Call Trace: <TASK> ? _raw_spin_unlock_irqrestore+0xa/0x30 ? __wake_up_common_lock+0x8a/0xc0 dbuf_sync_leaf+0x13e/0x550 [zfs] ? taskq_dispatch_ent+0x141/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common virtio_balloon pcspkr i2c_piix4 joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ata_piix ghash_clmulni_intel libata virtio_blk net_failover failover serio_raw [last unloaded: llog_test] CR2: 0000000000000018 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |