| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 0 PID: 888171 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE ------- --- 5.14.0-362.18.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffb5cbc68ef990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff9397f039c5a0 RCX: ffff9397e95bc780 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffb5cbc68efb08 RBP: ffffb5cbc68efa70 R08: ffffffffc0d9d290 R09: 0000000000000001 R10: ffffb5cbc68efa88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff9397efd87680 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff93983fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 0000000001970004 CR4: 00000000000606f0 Call Trace: <TASK> ? show_trace_log_lvl+0x1c4/0x2df ? show_trace_log_lvl+0x1c4/0x2df ? dbuf_write+0x304/0x540 [zfs] ? __die_body.cold+0x8/0xd ? page_fault_oops+0x134/0x170 ? kernelmode_fixup_or_oops+0x84/0x110 ? exc_page_fault+0x62/0x150 ? asm_exc_page_fault+0x22/0x30 ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? arc_write+0xf5/0x450 [zfs] ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? taskq_dispatch_ent+0x141/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? __pfx_default_wake_function+0x10/0x10 ? __pfx_taskq_thread+0x10/0x10 [spl] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) icp(POE) zcommon(POE) znvpair(POE) zavl(POE) spl(OE) zlua(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs sunrpc rfkill intel_rapl_msr intel_rapl_common virtio_balloon i2c_piix4 pcspkr joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 1 PID: 895717 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffb6a4c1a4b990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff9e3db5c9f460 RCX: ffff9e3d9abab980 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffb6a4c1a4bb08 RBP: ffffb6a4c1a4ba70 R08: ffffffffc0c54040 R09: 0000000000000001 R10: ffffb6a4c1a4ba88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff9e3dc68a5dd0 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff9e3dffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 0000000004dcc004 CR4: 00000000000606e0 Call Trace: <TASK> ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? zio_gang_tree_free+0x70/0x70 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common dm_mod i2c_piix4 joydev pcspkr virtio_balloon drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net net_failover ghash_clmulni_intel failover virtio_blk serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Autotest: Test running for 230 minutes (lustre-master_rolling-upgrade-client1-zfs_4516.164) | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 1 PID: 889811 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE ------- --- 5.14.0-362.18.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffa8c5c75eb990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff99e7f7f4efa0 RCX: ffff99e7f2e56c00 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffa8c5c75ebb08 RBP: ffffa8c5c75eba70 R08: ffffffffc0b73290 R09: 0000000000000001 R10: ffffa8c5c75eba88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff99e7f1bf2290 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff99e83fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 000000005fbc4005 CR4: 00000000001706e0 Call Trace: <TASK> ? show_trace_log_lvl+0x1c4/0x2df ? show_trace_log_lvl+0x1c4/0x2df ? dbuf_write+0x304/0x540 [zfs] ? __die_body.cold+0x8/0xd ? page_fault_oops+0x134/0x170 ? kernelmode_fixup_or_oops+0x84/0x110 ? exc_page_fault+0x62/0x150 ? asm_exc_page_fault+0x22/0x30 ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? arc_write+0xf5/0x450 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? __pfx_zio_execute+0x10/0x10 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? __pfx_default_wake_function+0x10/0x10 ? __pfx_taskq_thread+0x10/0x10 [spl] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) icp(POE) zcommon(POE) znvpair(POE) zavl(POE) spl(OE) zlua(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev dm_mod drm fuse ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net net_failover virtio_blk ghash_clmulni_intel failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 1 PID: 886000 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE ------- --- 5.14.0-362.18.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffff9e56c2fff990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff888725004640 RCX: ffff888703f7ef00 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff9e56c2fffb08 RBP: ffff9e56c2fffa70 R08: ffffffffc0e3c290 R09: 0000000000000001 R10: ffff9e56c2fffa88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff88871e9b6ca0 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88873fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 00000000348a4001 CR4: 00000000000606e0 Call Trace: <TASK> ? show_trace_log_lvl+0x1c4/0x2df ? show_trace_log_lvl+0x1c4/0x2df ? dbuf_write+0x304/0x540 [zfs] ? __die_body.cold+0x8/0xd ? page_fault_oops+0x134/0x170 ? kernelmode_fixup_or_oops+0x84/0x110 ? exc_page_fault+0x62/0x150 ? asm_exc_page_fault+0x22/0x30 ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? arc_write+0xf5/0x450 [zfs] ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? __pfx_dbuf_write_ready+0x10/0x10 [zfs] ? __pfx_dbuf_write_physdone+0x10/0x10 [zfs] ? __pfx_dbuf_write_done+0x10/0x10 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? __pfx_zio_execute+0x10/0x10 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? __pfx_default_wake_function+0x10/0x10 ? __pfx_taskq_thread+0x10/0x10 [spl] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) icp(POE) zcommon(POE) znvpair(POE) zavl(POE) spl(OE) zlua(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common virtio_balloon i2c_piix4 joydev dm_mod pcspkr fuse drm ext4 mbcache jbd2 ata_generic ata_piix libata crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 1 PID: 882191 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffa68e4238f990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff93190b7fc6e0 RCX: ffff9318fc80fb00 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffa68e4238fb08 RBP: ffffa68e4238fa70 R08: ffffffffc0d9e040 R09: 0000000000000001 R10: ffffa68e4238fa88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff9318855e93c0 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff93193fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 00000000353c0004 CR4: 00000000001706e0 Call Trace: <TASK> ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? zio_gang_tree_free+0x70/0x70 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill intel_rapl_msr intel_rapl_common pcspkr virtio_balloon i2c_piix4 joydev sunrpc fuse drm dm_mod ext4 mbcache jbd2 ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ata_piix libata net_failover ghash_clmulni_intel virtio_blk failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 1 PID: 887856 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffa27504eef990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff8b8f03a17730 RCX: ffff8b8efd791080 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffa27504eefb08 RBP: ffffa27504eefa70 R08: ffffffffc0b89040 R09: 0000000000000001 R10: ffffa27504eefa88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff8b8ef1d913c0 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8b8f3fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 0000000004258006 CR4: 00000000000606e0 Call Trace: <TASK> ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? taskq_dispatch_ent+0x141/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill intel_rapl_msr intel_rapl_common virtio_balloon sunrpc pcspkr i2c_piix4 dm_mod joydev drm fuse ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul libata crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel virtio_blk net_failover failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 0 PID: 888729 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffab59c23cb990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff9500c9496a50 RCX: ffff9500c4b3ca80 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffab59c23cbb08 RBP: ffffab59c23cba70 R08: ffffffffc0c8f040 R09: 0000000000000001 R10: ffffab59c23cba88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff9500a7793b40 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff9500ffc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 00000000044f4005 CR4: 00000000000606f0 Call Trace: <TASK> ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? zio_gang_tree_free+0x70/0x70 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common virtio_balloon pcspkr i2c_piix4 joydev dm_mod fuse drm ext4 mbcache jbd2 ata_generic ata_piix crct10dif_pclmul libata crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel net_failover virtio_blk failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 0 PID: 770122 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffad5044f23990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff8c7f23081000 RCX: ffff8c7f3935c780 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffad5044f23b08 RBP: ffffad5044f23a70 R08: ffffffffc0b37040 R09: 0000000000000001 R10: ffffad5044f23a88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff8c7f1a87b160 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8c7f7fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 00000000180de006 CR4: 00000000000606f0 Call Trace: <TASK> ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? taskq_dispatch_ent+0x141/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common i2c_piix4 dm_mod pcspkr virtio_balloon joydev fuse drm ext4 mbcache jbd2 ata_generic virtio_net crct10dif_pclmul ata_piix crc32_pclmul crc32c_intel net_failover libata failover ghash_clmulni_intel virtio_blk serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |
| sanity test 136: Race catalog processing 2 | BUG: kernel NULL pointer dereference, address: 0000000000000028 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 0 P4D 0 Oops: 0002 [#1] PREEMPT SMP PTI CPU: 0 PID: 760577 Comm: dp_sync_taskq Kdump: loaded Tainted: P W OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:arc_write+0xf5/0x450 [zfs] Code: 28 48 89 55 a0 48 8b 50 30 48 89 55 a8 48 8b 50 38 48 89 55 b0 48 8b 50 40 48 8b 40 48 48 89 55 b8 48 89 45 c0 45 85 c9 74 05 <41> 83 4f 28 10 8b 43 48 a8 04 0f 84 4b 02 00 00 b8 00 00 00 01 b9 RSP: 0018:ffffbd0301d73990 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff9b9804db3eb0 RCX: ffff9b97e0c61380 RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffffbd0301d73b08 RBP: ffffbd0301d73a70 R08: ffffffffc0e3c040 R09: 0000000000000001 R10: ffffbd0301d73a88 R11: 0000000000000007 R12: 0000000000000000 R13: ffff9b97fab253f0 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff9b983fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000028 CR3: 000000000396a003 CR4: 00000000000606f0 Call Trace: <TASK> ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_write+0x304/0x540 [zfs] dbuf_write+0x304/0x540 [zfs] ? dmu_buf_unlock_parent+0x70/0x70 [zfs] ? dbuf_unoverride+0x90/0x90 [zfs] ? dmu_buf_rele+0x40/0x40 [zfs] ? _raw_spin_unlock+0xa/0x30 ? dbuf_hold_impl+0x104/0x6e0 [zfs] dbuf_sync_leaf+0x13e/0x550 [zfs] ? zio_gang_tree_free+0x70/0x70 [zfs] ? taskq_dispatch_ent+0x107/0x1c0 [spl] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? zio_issue_async+0x4f/0x90 [zfs] ? zio_nowait+0xb4/0x1a0 [zfs] dbuf_sync_list+0xc1/0x120 [zfs] dbuf_sync_indirect+0xe3/0x170 [zfs] dbuf_sync_list+0x50/0x120 [zfs] dnode_sync+0x3e3/0x750 [zfs] ? dnode_multilist_index_func+0x94/0xb0 [zfs] sync_dnodes_task+0x71/0xa0 [zfs] taskq_thread+0x2cf/0x500 [spl] ? wake_up_q+0x90/0x90 ? taskq_thread_spawn+0x60/0x60 [spl] kthread+0xd9/0x100 ? kthread_complete_and_exit+0x20/0x20 ret_from_fork+0x22/0x30 </TASK> Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(POE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill intel_rapl_msr intel_rapl_common i2c_piix4 virtio_balloon pcspkr sunrpc dm_mod joydev fuse drm ext4 mbcache jbd2 ata_generic crct10dif_pclmul ata_piix crc32_pclmul crc32c_intel libata virtio_net ghash_clmulni_intel net_failover virtio_blk failover serio_raw [last unloaded: llog_test] CR2: 0000000000000028 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 | Link to test |