| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| replay-single test complete, duration 274 sec | BUG: unable to handle kernel NULL pointer dereference at 00000000000000c0 PGD 0 P4D 0 Oops: 0002 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 6 Comm: kworker/0:0 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events 0xffffffffc0e72750 RIP: 0010:down_write+0x57/0xd0 Code: cf 91 ff ff 48 83 05 37 f4 f9 01 01 48 83 05 9f ee f9 01 01 65 ff 05 00 b3 04 48 48 83 05 98 ee f9 01 01 31 c0 ba 01 00 00 00 <f0> 48 0f b1 13 0f 94 c0 48 83 05 89 ee f9 01 01 84 c0 75 3e be 02 RSP: 0018:ffffb05cc0c87d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 00000000000000c0 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000626 RDI: ffffffffb89869d4 RBP: ffff88bfacf48948 R08: 0000000000000000 R09: 0000000000000000 R10: ffff88bfb5f52880 R11: 00000000fffffff8 R12: ffff88bfacf489a0 R13: ffff88bfacf48a00 R14: ffffffffba8587d4 R15: ffff88bfacf48948 FS: 0000000000000000(0000) GS:ffff88bfc2000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000000c0 CR3: 0000000050e16004 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? down_write+0x57/0xd0 ? down_write+0x31/0xd0 debugfs_remove_recursive+0x59/0x260 ? mutex_lock+0x31/0x70 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] ? process_one_work+0x2c8/0x700 ? worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Modules linked in: zfs(O) spl(O) ldiskfs(O) lquota(O) ec(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: ptlrpc_gss] CR2: 00000000000000c0 | Lustre: DEBUG MARKER: === replay-single: start cleanup 23:10:46 (1779419446) === Lustre: DEBUG MARKER: === replay-single: finish cleanup 23:10:48 (1779419448) === LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 6596:0:(ldlm_lib.c:1182:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 6596:0:(ldlm_lib.c:1182:target_handle_connect()) Skipped 12 previous similar messages LustreError: 6576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779419457 with bad export cookie 7080792776186128427 LustreError: MGC192.168.202.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 6576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg258-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |
| replay-single test complete, duration 241 sec | BUG: unable to handle kernel NULL pointer dereference at 00000000000000c0 PGD 0 P4D 0 Oops: 0002 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 6 Comm: kworker/0:0 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events 0xffffffffc18e2750 RIP: 0010:down_write+0x57/0xd0 Code: cf 91 ff ff 48 83 05 37 f4 f9 01 01 48 83 05 9f ee f9 01 01 65 ff 05 00 b3 24 48 48 83 05 98 ee f9 01 01 31 c0 ba 01 00 00 00 <f0> 48 0f b1 13 0f 94 c0 48 83 05 89 ee f9 01 01 84 c0 75 3e be 02 RSP: 0018:ffffb36240c87d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 00000000000000c0 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000626 RDI: ffffffffb87869d4 RBP: ffff90ca057815e8 R08: 0000000000000000 R09: 0000000000000000 R10: ffff90c906116b40 R11: 00000000ffffffe2 R12: ffff90ca05781640 R13: ffff90ca057816a0 R14: ffffffffba6587d4 R15: ffff90ca057815e8 FS: 0000000000000000(0000) GS:ffff90ca42000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000000c0 CR3: 00000000ab216005 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? down_write+0x57/0xd0 ? down_write+0x31/0xd0 debugfs_remove_recursive+0x59/0x260 ? mutex_lock+0x31/0x70 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] ? process_one_work+0x2c8/0x700 ? worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Modules linked in: lquota(O) ec(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul ata_generic crc32c_intel ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: ptlrpc_gss] CR2: 00000000000000c0 | Lustre: DEBUG MARKER: === replay-single: start cleanup 23:09:45 (1779419385) === Lustre: DEBUG MARKER: === replay-single: finish cleanup 23:09:46 (1779419386) === LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 9202:0:(ldlm_lib.c:1182:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 9202:0:(ldlm_lib.c:1182:target_handle_connect()) Skipped 3 previous similar messages LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779419401 with bad export cookie 12782072568302625087 LustreError: MGC192.168.204.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg415-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |
| replay-single test complete, duration 228 sec | BUG: unable to handle kernel NULL pointer dereference at 00000000000000c0 PGD 0 P4D 0 Oops: 0002 [#1] SMP DEBUG_PAGEALLOC CPU: 2 PID: 799 Comm: kworker/2:3 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events 0xffffffffc18f8750 RIP: 0010:down_write+0x57/0xd0 Code: cf 91 ff ff 48 83 05 37 f4 f9 01 01 48 83 05 9f ee f9 01 01 65 ff 05 00 b3 44 7d 48 83 05 98 ee f9 01 01 31 c0 ba 01 00 00 00 <f0> 48 0f b1 13 0f 94 c0 48 83 05 89 ee f9 01 01 84 c0 75 3e be 02 RSP: 0018:ffffac9400f6fd98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 00000000000000c0 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000626 RDI: ffffffff835869d4 RBP: ffff969711297bd0 R08: 0000000000000000 R09: 0000000000000000 R10: ffff969707de3080 R11: 0000000000000007 R12: ffff969711297c28 R13: ffff969711297c88 R14: ffffffff854587d4 R15: ffff969711297bd0 FS: 0000000000000000(0000) GS:ffff969742100000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000000000c0 CR3: 0000000018e16002 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? 0xffffffff81c00000 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? down_write+0x57/0xd0 ? down_write+0x31/0xd0 debugfs_remove_recursive+0x59/0x260 ? mutex_lock+0x31/0x70 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] ? process_one_work+0x2c8/0x700 ? worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Modules linked in: lquota(O) ec(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: ptlrpc_gss] CR2: 00000000000000c0 | Lustre: DEBUG MARKER: === replay-single: start cleanup 23:09:13 (1779419353) === Lustre: DEBUG MARKER: === replay-single: finish cleanup 23:09:15 (1779419355) === LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 6489:0:(ldlm_lib.c:1182:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 6489:0:(ldlm_lib.c:1182:target_handle_connect()) Skipped 3 previous similar messages LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779419367 with bad export cookie 8043050266150454617 LustreError: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |