| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| replay-single test complete, duration 2668 sec | BUG: unable to handle kernel paging request at ffff8c6f9cd1d978 PGD 69201067 P4D 69201067 PUD 146dcf067 PMD 1465e7067 PTE 800ffffee32e2060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 90194 Comm: kworker/0:3 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events rsc_entry_free_work [ptlrpc_gss] RIP: 0010:debugfs_remove_recursive+0x45/0x260 Code: 41 57 49 89 ff 41 56 49 c7 c6 d4 87 05 a1 41 55 4c 8d af b8 00 00 00 41 54 55 48 89 fd 53 48 83 ec 08 48 83 05 eb 9a 17 03 01 <48> 8b 45 30 4c 8d 65 58 48 8d b8 c0 00 00 00 e8 c7 3e 8f 00 48 83 RSP: 0018:ffffa5c1c2c3bda8 EFLAGS: 00010202 RAX: 0000000000000001 RBX: ffff8c6f88502280 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8c6f83643c2c RDI: ffff8c6f9cd1d948 RBP: ffff8c6f9cd1d948 R08: 0000000000000000 R09: 0000000000000000 R10: ffff8c6e84608360 R11: 00000000fffffffb R12: ffff8c6fc2038a00 R13: ffff8c6f9cd1da00 R14: ffffffffa10587d4 R15: ffff8c6f9cd1d948 FS: 0000000000000000(0000) GS:ffff8c6fc2000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff8c6f9cd1d978 CR3: 0000000066e16001 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_kern_addr_fault+0x73/0x130 ? __do_page_fault+0x8e/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? debugfs_remove_recursive+0x45/0x260 ? _cond_resched+0x1d/0x50 ? mutex_lock+0x31/0x70 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] gssiam_delete_sec_context+0x22/0x80 [ptlrpc_gss] lgss_delete_sec_context+0x60/0x2d0 [ptlrpc_gss] __rsc_free+0x9b/0x140 [ptlrpc_gss] rsc_entry_free_work+0x21/0xf0 [ptlrpc_gss] process_one_work+0x2c8/0x700 worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: zfs(O) spl(O) ldiskfs(O) lquota(O) ec(O) ptlrpc_gss(O-) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs crct10dif_pclmul crc32_pclmul ata_generic crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: mgc] CR2: ffff8c6f9cd1d978 | Lustre: DEBUG MARKER: === replay-single: start cleanup 02:50:57 (1779432657) === Lustre: DEBUG MARKER: === replay-single: finish cleanup 02:50:58 (1779432658) === Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2051 to 0x280000401:2145) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2019 to 0x2c0000401:2113) Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 6593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779432692 with bad export cookie 14302943742898896452 LustreError: 6593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |
| replay-single test complete, duration 157 sec | BUG: unable to handle kernel paging request at 0000000a00000058 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 2 PID: 1015 Comm: kworker/2:4 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events rsc_entry_free_work [ptlrpc_gss] RIP: 0010:debugfs_remove_recursive+0x45/0x260 Code: 41 57 49 89 ff 41 56 49 c7 c6 d4 87 a5 99 41 55 4c 8d af b8 00 00 00 41 54 55 48 89 fd 53 48 83 ec 08 48 83 05 eb 9a 17 03 01 <48> 8b 45 30 4c 8d 65 58 48 8d b8 c0 00 00 00 e8 c7 3e 8f 00 48 83 RSP: 0018:ffffad5001847da8 EFLAGS: 00010206 RAX: 0000000000000001 RBX: ffff8fd233087cc0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8fd108093c2c RDI: 0000000a00000028 RBP: 0000000a00000028 R08: 0000000000000000 R09: 0000000000000000 R10: ffff8fd232b51c00 R11: 00000000fffffff9 R12: ffff8fd242138a00 R13: 0000000a000000e0 R14: ffffffff99a587d4 R15: 0000000a00000028 FS: 0000000000000000(0000) GS:ffff8fd242100000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000a00000058 CR3: 000000013ae16005 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __free_pages+0x102/0x160 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? debugfs_remove_recursive+0x45/0x260 ? _cond_resched+0x1d/0x50 ? mutex_lock+0x31/0x70 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] gssiam_delete_sec_context+0x22/0x80 [ptlrpc_gss] lgss_delete_sec_context+0x60/0x2d0 [ptlrpc_gss] __rsc_free+0x9b/0x140 [ptlrpc_gss] rsc_entry_free_work+0x21/0xf0 [ptlrpc_gss] process_one_work+0x2c8/0x700 worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lquota(O) ec(O) ptlrpc_gss(O-) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: mgc] CR2: 0000000a00000058 | Lustre: DEBUG MARKER: === replay-single: start cleanup 02:15:14 (1779430514) === Lustre: DEBUG MARKER: === replay-single: finish cleanup 02:15:15 (1779430515) === LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 6425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779430521 with bad export cookie 14491639712430793476 LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 6425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |
| recovery-small test complete, duration 393 sec | BUG: unable to handle kernel paging request at 0000000500000058 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 22698 Comm: kworker/0:0 Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Workqueue: events rsc_entry_free_work [ptlrpc_gss] RIP: 0010:debugfs_remove_recursive+0x45/0x260 Code: 41 57 49 89 ff 41 56 49 c7 c6 d4 87 05 aa 41 55 4c 8d af b8 00 00 00 41 54 55 48 89 fd 53 48 83 ec 08 48 83 05 eb 9a 17 03 01 <48> 8b 45 30 4c 8d 65 58 48 8d b8 c0 00 00 00 e8 c7 3e 8f 00 48 83 RSP: 0018:ffffab6504a6bda8 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff9ad803c9e580 RCX: 0000000007d682c5 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000500000028 RBP: 0000000500000028 R08: 00000000000004bb R09: 0000000000000000 R10: 00000000024acb51 R11: ffff9ad90990c000 R12: ffff9ad942038a00 R13: 00000005000000e0 R14: ffffffffaa0587d4 R15: 0000000500000028 FS: 0000000000000000(0000) GS:ffff9ad942000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000500000058 CR3: 00000000b2016005 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? snprintf+0x51/0x70 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? debugfs_remove_recursive+0x45/0x260 lprocfs_nodemap_remove+0x1a/0x160 [ptlrpc] nodemap_destroy+0x45/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] nodemap_destroy+0x298/0x700 [ptlrpc] nodemap_putref+0xd3/0x130 [ptlrpc] gssiam_delete_sec_context+0x22/0x80 [ptlrpc_gss] lgss_delete_sec_context+0x60/0x2d0 [ptlrpc_gss] __rsc_free+0x9b/0x140 [ptlrpc_gss] rsc_entry_free_work+0x21/0xf0 [ptlrpc_gss] process_one_work+0x2c8/0x700 worker_thread+0x296/0x6e0 ? rescuer_thread+0x570/0x570 kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lquota(O) ec(O) ptlrpc_gss(O-) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic serio_raw ghash_clmulni_intel ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: mgc] CR2: 0000000500000058 | Lustre: DEBUG MARKER: === recovery-small: start cleanup 02:13:32 (1779430412) === Lustre: DEBUG MARKER: === recovery-small: finish cleanup 02:13:33 (1779430413) === Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 17767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779430422 with bad export cookie 1890942987389911723 LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 17767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local Key type lgssc unregistered | Link to test |