| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-scrub test 13: OI scrub can rebuild missed /O entries | BUG: unable to handle kernel paging request at ffffffffc1cd7f20 PGD 32e18067 P4D 32e18067 PUD 32e1a067 PMD 105fa8067 PTE 140677161 Oops: 0003 [#1] SMP DEBUG_PAGEALLOC CPU: 3 PID: 57313 Comm: OI_scrub Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 RIP: 0010:mutex_lock+0x44/0x70 Code: 01 04 fa 01 01 e8 1c 52 19 ff 48 83 05 fc 03 fa 01 01 e8 ef c6 ff ff 48 83 05 f7 03 fa 01 01 31 c0 65 48 8b 14 25 00 dd 01 00 <f0> 48 0f b1 13 74 1a 48 89 df 48 83 05 ea 03 fa 01 01 e8 75 ff ff RSP: 0018:ffffc42e42bc7cb8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffffffc1cd7f20 RCX: 00000000000125e0 RDX: ffffa05d4a6aa7c0 RSI: 0000000000000121 RDI: ffffffff9cf86790 RBP: 81660ce2c104eac0 R08: 0000000000000029 R09: 000000000000001a R10: ffffa05d7a656300 R11: f000000000000000 R12: ffffffffc1cd7f20 R13: 0000000000000200 R14: ffffa05d7e3c0000 R15: ffffa05d716f0000 FS: 0000000000000000(0000) GS:ffffa05d82180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffc1cd7f20 CR3: 0000000032e16003 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? no_context+0x30f/0x5a0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? bad_area_nosemaphore+0x1a/0x30 ? do_kern_addr_fault+0x73/0x130 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __do_page_fault+0x8e/0xa0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? mutex_lock+0x44/0x70 ? mutex_lock+0x31/0x70 ? osd_scan_O_seq+0x680/0x680 [osd_zfs] sa_handle_destroy+0x2a/0x170 [zfs] osd_create_lastid+0x1c4/0x920 [osd_zfs] ? osd_scan_O_seq+0x680/0x680 [osd_zfs] ? osd_scan_dir+0x2a2/0x620 [osd_zfs] osd_scan_lastid_seq+0x6c5/0x11e0 [osd_zfs] ? osd_create_lastid+0x920/0x920 [osd_zfs] osd_scan_dir+0x185/0x620 [osd_zfs] osd_scrub_main+0x947/0x1130 [osd_zfs] ? init_wait_var_entry+0x50/0x50 ? osd_scrub_check_update+0xf90/0xf90 [osd_zfs] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) ec(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: ffffffffc1cd7f20 | Lustre: *** cfs_fail_loc=196, val=0*** Lustre: Skipped 63 previous similar messages Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages LustreError: 32085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 37784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 56746:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 6 previous similar messages Lustre: DEBUG MARKER: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: 57313:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] LustreError: 57313:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) Skipped 2 previous similar messages | Link to test (2026-09-03 08:33) |
| sanity-scrub test 13: OI scrub can rebuild missed /O entries | BUG: unable to handle kernel paging request at ffffffffc1f71fc0 PGD 66a18067 P4D 66a18067 PUD 66a1a067 PMD 105317067 PTE 12da7c161 Oops: 0003 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 55647 Comm: OI_scrub Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 RIP: 0010:mutex_lock+0x44/0x70 Code: 01 04 fa 01 01 e8 1c 52 19 ff 48 83 05 fc 03 fa 01 01 e8 ef c6 ff ff 48 83 05 f7 03 fa 01 01 31 c0 65 48 8b 14 25 00 dd 01 00 <f0> 48 0f b1 13 74 1a 48 89 df 48 83 05 ea 03 fa 01 01 e8 75 ff ff RSP: 0018:ffff9dbd817c3cb8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffffffc1f71fc0 RCX: 00000000000120cb RDX: ffff90c58873a7c0 RSI: 0000000000000121 RDI: ffffffffaf186790 RBP: 81660ce2c104eac0 R08: 0000000000000029 R09: 0000000000000005 R10: ffff90c689682d80 R11: f000000000000000 R12: ffffffffc1f71fc0 R13: 0000000000000200 R14: ffff90c6bfb80000 R15: ffff90c6adf21800 FS: 0000000000000000(0000) GS:ffff90c6c2000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffc1f71fc0 CR3: 0000000066a16003 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? no_context+0x30f/0x5a0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? bad_area_nosemaphore+0x1a/0x30 ? do_kern_addr_fault+0x73/0x130 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __do_page_fault+0x8e/0xa0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? mutex_lock+0x44/0x70 ? mutex_lock+0x31/0x70 ? osd_scan_O_seq+0x670/0x670 [osd_zfs] sa_handle_destroy+0x2a/0x170 [zfs] osd_create_lastid+0x1c4/0x920 [osd_zfs] ? osd_scan_O_seq+0x670/0x670 [osd_zfs] ? osd_scan_dir+0x2a2/0x620 [osd_zfs] osd_scan_lastid_seq+0x6c5/0x11e0 [osd_zfs] ? osd_create_lastid+0x920/0x920 [osd_zfs] osd_scan_dir+0x185/0x620 [osd_zfs] osd_scrub_main+0x947/0x1130 [osd_zfs] ? init_wait_var_entry+0x50/0x50 ? osd_scrub_check_update+0xf90/0xf90 [osd_zfs] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) ec(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul ata_generic crc32_pclmul crc32c_intel ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: ffffffffc1f71fc0 | Lustre: *** cfs_fail_loc=196, val=0*** Lustre: Skipped 63 previous similar messages Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 6 previous similar messages LustreError: 41639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 41636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 55083:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 6 previous similar messages Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: 55647:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] LustreError: 55647:0:(osd_oi.c:544:osd_get_idx_for_ost_obj()) Skipped 2 previous similar messages | Link to test (2026-08-15 18:51) |
| sanity-scrub test 13: OI scrub can rebuild missed /O entries | BUG: unable to handle kernel paging request at ffffffffc2036de0 PGD 11f818067 P4D 11f818067 PUD 11f81a067 PMD 105d3b067 PTE 14019e161 Oops: 0003 [#1] SMP DEBUG_PAGEALLOC CPU: 2 PID: 55617 Comm: OI_scrub Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 RIP: 0010:mutex_lock+0x44/0x70 Code: 01 04 fa 01 01 e8 1c 52 19 ff 48 83 05 fc 03 fa 01 01 e8 ef c6 ff ff 48 83 05 f7 03 fa 01 01 31 c0 65 48 8b 14 25 00 dd 01 00 <f0> 48 0f b1 13 74 1a 48 89 df 48 83 05 ea 03 fa 01 01 e8 75 ff ff RSP: 0018:ffffbac3c4ed7cb8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffffffffc2036de0 RCX: 00000000000139ff RDX: ffff9e712d68cf80 RSI: 0000000000000121 RDI: ffffffff88986790 RBP: 81660ce2c104eac0 R08: 0000000000000000 R09: 9ae16a3b2f90404f R10: ffff9e71109a7b40 R11: f000000000000000 R12: ffffffffc2036de0 R13: 0000000000000200 R14: ffff9e71064c0000 R15: ffff9e711a8f9000 FS: 0000000000000000(0000) GS:ffff9e7142100000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffc2036de0 CR3: 000000011f816003 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? no_context+0x30f/0x5a0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? bad_area_nosemaphore+0x1a/0x30 ? do_kern_addr_fault+0x73/0x130 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? __do_page_fault+0x8e/0xa0 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? osd_scan_lastid_dir+0xe0/0x300 [osd_zfs] ? mutex_lock+0x44/0x70 ? mutex_lock+0x31/0x70 ? osd_scan_O_seq+0x670/0x670 [osd_zfs] sa_handle_destroy+0x2a/0x170 [zfs] osd_create_lastid+0x1c4/0x920 [osd_zfs] ? osd_scan_O_seq+0x670/0x670 [osd_zfs] ? osd_scan_dir+0x2a8/0x630 [osd_zfs] osd_scan_lastid_seq+0x6c5/0x11e0 [osd_zfs] ? osd_create_lastid+0x920/0x920 [osd_zfs] osd_scan_dir+0x18b/0x630 [osd_zfs] osd_scrub_main+0x947/0x1130 [osd_zfs] ? init_wait_var_entry+0x50/0x50 ? osd_scrub_check_update+0xf90/0xf90 [osd_zfs] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) ec(O) zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: ffffffffc2036de0 | Lustre: *** cfs_fail_loc=196, val=0*** Lustre: Skipped 63 previous similar messages Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages LustreError: 36822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 36823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 55039:0:(osd_oi.c:541:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 6 previous similar messages Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: 55617:0:(osd_oi.c:541:osd_get_idx_for_ost_obj()) lustre-OST0000: Can not find seq group [0x200000003:0x0:0x0] LustreError: 55617:0:(osd_oi.c:541:osd_get_idx_for_ost_obj()) Skipped 2 previous similar messages | Link to test (2026-07-14 19:12) |