| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 140498:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 140498:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 140498 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f76d12408fb | Lustre: DEBUG MARKER: Started rundbench load pid=140317 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 137869:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 0000000098ea04d0 ns: mdt-lustre-MDT0000_UUID lock: 000000009857aaee/0x21f011dcce788409 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf20:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x21f011dcce7883fb expref: 3 pid: 137869 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-25 00:29) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 816002:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 815605:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 000000005ecd82f9 ns: mdt-lustre-MDT0000_UUID lock: 0000000099ce2ad8/0x150b1667cd4bee75 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf07:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x150b1667cd4bee67 expref: 3 pid: 815605 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 LustreError: 816002:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 816002 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] Lustre: Skipped 4 previous similar messages ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fdb0c6cf8fb | Lustre: DEBUG MARKER: Started rundbench load pid=814313 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3914 to 0x240000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3915 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3958 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 815013:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000a9454c10 ns: mdt-lustre-MDT0000_UUID lock: 00000000e07e302b/0x150b1667cd4b4e4e lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xf0e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x150b1667cd4b4e40 expref: 3 pid: 815013 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4018 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3953 to 0x240000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3953 to 0x280000400:3969) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 | Link to test (2026-10-23 09:43) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1816623:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1816623:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 1816623 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x56/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] LustreError: 1795577:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 1 previous similar message deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f6d7017d8fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff91904aeda800: operation mds_reint to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1215 to 0x280000400:1249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1213 to 0x240000400:1249) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1213 to 0x2c0000400:1249) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1213 to 0x300000400:1249) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1318 to 0x280000400:1345) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1319 to 0x2c0000400:1345) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1317 to 0x240000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1318 to 0x300000400:1345) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 1795581:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9190a939b480 x1879535942013184/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1411 to 0x240000400:1441) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1411 to 0x2c0000400:1441) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1410 to 0x300000400:1441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1411 to 0x280000400:1441) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 1795576:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792466550/real 1792466550] req@ffff919092dfa300 x1879535942831488/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1792466566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 1795576:0:(client.c:2514:ptlrpc_expire_one_request()) Skipped 67 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1492 to 0x280000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1491 to 0x2c0000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1492 to 0x240000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1492 to 0x300000400:1537) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 1795577:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91908ea26c80 x1879535943693184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1816339:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000f227276d ns: mdt-lustre-MDT0000_UUID lock: 000000006a043c3c/0x87b9a98a067b6187 lrc: 4/0,0 mode: PR/PR res: [0x20000afe1:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x87b9a98a067b6179 expref: 61 pid: 1816339 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-20 03:23) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 2469033:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2469033:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 2469033 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f23143b88fb | Lustre: DEBUG MARKER: Started rundbench load pid=2460638 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3913 to 0x240000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3958 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3952 to 0x300000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4045 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3958 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3948 to 0x240000400:3969) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4017 to 0x2c0000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4087 to 0x280000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3992 to 0x300000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4002 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 2462693:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000444117c3 ns: mdt-lustre-MDT0000_UUID lock: 000000004dc26238/0x13add0f0fb9a1d97 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xef5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x13add0f0fb9a1d89 expref: 3 pid: 2462693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 38 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4058 to 0x300000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4154 to 0x280000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4057 to 0x2c0000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4058 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4111 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4208 to 0x280000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4106 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4118 to 0x240000400:4161) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4150 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4156 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4183 to 0x240000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4244 to 0x280000400:4289) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 2392530:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792211756/real 1792211756] req@ffff9ebadf7a5780 x1879267371082880/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1792211772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 2392530:0:(client.c:2514:ptlrpc_expire_one_request()) Skipped 75 previous similar messages LustreError: MGC192.168.123.88@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4214 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4214 to 0x2c0000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4247 to 0x240000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4311 to 0x280000400:4353) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4308 to 0x240000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4275 to 0x2c0000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4372 to 0x280000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4276 to 0x300000400:4321) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff9ebad4226800: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4342 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4379 to 0x240000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4444 to 0x280000400:4481) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4338 to 0x300000400:4353) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 2467506:0:(osd_handler.c:724:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 2467506:0:(osd_handler.c:724:osd_ro()) Skipped 9 previous similar messages Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 2392530:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ebaec82b100 x1879267373338368/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 LustreError: 2392530:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 2392526:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9ebae042d400 x1879267368268928/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff9ebad4226800@0@lo:12/10 lens 576/608 e 0 to 0 dl 1792211884 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 2392526:0:(client.c:3445:ptlrpc_replay_interpret()) Skipped 356 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4495 to 0x280000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4366 to 0x300000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4435 to 0x240000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4401 to 0x2c0000400:4417) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4408 to 0x300000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4524 to 0x280000400:4545) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4505 to 0x240000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4438 to 0x2c0000400:4481) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: 2468673:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000a29a325c ns: mdt-lustre-MDT0000_UUID lock: 000000005866b85f/0x13add0f0fb9ee2ed lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xeed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x13add0f0fb9ee2df expref: 3 pid: 2468673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-17 04:38) |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | LustreError: 3422753:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3422753:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 3422753 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fd9e1d058fb | Lustre: DEBUG MARKER: Started rundbench load pid=3414633 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 3414748:0:(osd_handler.c:724:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 3414748:0:(osd_handler.c:724:osd_ro()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages LustreError: 3348456:0:(ldlm_lib.c:1278:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 3348456:0:(ldlm_lib.c:1278:target_handle_connect()) Skipped 124 previous similar messages LustreError: MGC192.168.123.78@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a650d665400 x1879119871318656/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8a6491973000@0@lo:12/10 lens 576/608 e 0 to 0 dl 1792071478 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3748 to 0x300000402:3777) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3745) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3745) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3706 to 0x2c0000402:3745) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -107 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -19 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -19 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -19 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -11 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 2 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000400:204 to 0x300000400:225) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:203 to 0x2c0000400:225) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000400:203 to 0x380000400:257) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000400:202 to 0x340000400:225) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a64d4c6aa00 x1879119871327744/t4294967422(4294967422) o101->lustre-MDT0002-mdc-ffff8a6491973000@0@lo:12/10 lens 624/608 e 0 to 0 dl 1792071526 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000401:183 to 0x300000401:225) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000401:155 to 0x340000401:193) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000401:180 to 0x2c0000401:225) Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000401:173 to 0x380000401:193) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a650d665400 x1879119871318656/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8a6491973000@0@lo:12/10 lens 576/608 e 0 to 0 dl 1792071551 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) Skipped 22 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3777) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3777) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3748 to 0x300000402:3809) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3706 to 0x2c0000402:3777) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 56 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -19 Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0001-lwp-OST0003: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 61 previous similar messages Lustre: lustre-MDT0001: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000400:203 to 0x380000400:289) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:203 to 0x2c0000400:257) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000400:202 to 0x340000400:257) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000400:204 to 0x300000400:257) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a64d4c6aa00 x1879119871327744/t4294967422(4294967422) o101->lustre-MDT0002-mdc-ffff8a6491973000@0@lo:12/10 lens 624/608 e 0 to 0 dl 1792071600 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000401:393 to 0x300000401:417) Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000401:362 to 0x380000401:385) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000401:362 to 0x340000401:385) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3809) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3809) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3706 to 0x2c0000402:3809) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3748 to 0x300000402:3841) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -19 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000400:203 to 0x380000400:321) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:203 to 0x2c0000400:289) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000400:202 to 0x340000400:289) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000400:204 to 0x300000400:289) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8a64d4c6aa00 x1879119871327744/t4294967422(4294967422) o101->lustre-MDT0002-mdc-ffff8a6491973000@0@lo:12/10 lens 624/608 e 0 to 0 dl 1792071673 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3342495:0:(client.c:3445:ptlrpc_replay_interpret()) Skipped 38 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000401:594 to 0x300000401:609) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000401:561 to 0x340000401:577) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000401:594 to 0x2c0000401:609) Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000401:561 to 0x380000401:577) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3841) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3841) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3706 to 0x2c0000402:3841) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3748 to 0x300000402:3873) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 11 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8a6491973000: can't stat MDS #0: rc = -107 LustreError: 3414671:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 6 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:203 to 0x2c0000400:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000400:203 to 0x380000400:353) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000400:202 to 0x340000400:321) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000400:204 to 0x300000400:321) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 12 times Lustre: Failing over lustre-MDT0002 LustreError: 3346268:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000877449b1 ns: mdt-lustre-MDT0002_UUID lock: 00000000fcd35603/0x1f13316c1a5e80aa lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0xaf3:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x1f13316c1a5e809c expref: 3 pid: 3346268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-15 13:42) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1083567:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1083567:0:(ldlm_lockd.c:1010:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 1083567 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x62c/0xd60 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdbd/0x2510 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x10bb/0x1c80 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fd7ebe218fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff995d8e59b800: operation mds_reint to node 0@lo failed: rc = -19 LustreError: 1077751:0:(ldlm_lib.c:1278:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete LustreError: 1054355:0:(client.c:3445:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff995e09555780 x1879030191773568/t94489280710(94489280710) o101->lustre-MDT0000-mdc-ffff995d8e59b800@0@lo:12/10 lens 576/608 e 0 to 0 dl 1791984225 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 1054355:0:(client.c:3445:ptlrpc_replay_interpret()) Skipped 2 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1223 to 0x300000400:1249) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1223 to 0x2c0000400:1249) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1224 to 0x240000400:1249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1223 to 0x280000400:1249) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 28 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1305 to 0x300000400:1345) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1305 to 0x280000400:1345) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1305 to 0x2c0000400:1345) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1305 to 0x240000400:1345) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 1054367:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff995dea86bf00 x1879030193822976/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1408 to 0x2c0000400:1441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1407 to 0x240000400:1441) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1407 to 0x300000400:1441) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1407 to 0x280000400:1441) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 ODEBUG: object 00000000c5d3897a is on stack 00000000a27642d7, but NOT annotated. WARNING: CPU: 3 PID: 1081469 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) crc32_generic zfs(O) spl(O) i2c_piix4 virtio_balloon pcspkr rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 CPU: 3 PID: 1081469 Comm: mdt00_000 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 RIP: 0010:__debug_object_init.cold.5+0x35/0x158 Code: 9e b5 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff RSP: 0018:ffffb628c8c4b328 EFLAGS: 00010006 RAX: 0000000000000050 RBX: ffffb628c8c4b470 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff995fb22de5a8 RDI: ffff995fb22de5a8 RBP: ffffb628c8c4b378 R08: 0000000000000000 R09: c0000000ffff7fff R10: 0000000000000001 R11: ffffb628c8c4b0f8 R12: ffffffffb74c04e8 R13: ffffffffb6105aa0 R14: 000000000003b120 R15: ffff995d851be708 FS: 0000000000000000(0000) GS:ffff995fb22c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8936a7c860 CR3: 00000001ba43e000 CR4: 00000000000006e0 Call Trace: ? show_regs.cold.10+0x22/0x2f ? __warn+0xd0/0x160 ? __debug_object_init.cold.5+0x35/0x158 ? report_bug+0x120/0x150 ? __irq_work_queue_local+0xc7/0x310 ? do_error_trap+0xc3/0x140 ? do_invalid_op+0x4b/0x60 ? __debug_object_init.cold.5+0x35/0x158 ? invalid_op+0x14/0x20 ? __debug_object_init.cold.5+0x35/0x158 ? __debug_object_init.cold.5+0x74/0x158 ? lod_comp_is_ost_used+0x90/0x90 [lod] debug_object_init+0x26/0x30 init_timer_key+0x2d/0x120 lod_ost_alloc_qos+0x110c/0x2620 [lod] ? cfs_trace_unlock_tcd+0x2c/0xb0 [libcfs] ? libcfs_debug_msg+0xec9/0x13f0 [libcfs] lod_comp_prep_create+0x1780/0x1c30 [lod] lod_prepare_create+0x202/0x450 [lod] lod_declare_striped_create+0x270/0xd40 [lod] ? lod_sub_declare_create+0xcc/0x150 [lod] lod_declare_create+0x237/0xc30 [lod] dt_declare_create+0x79/0x230 [obdclass] mdd_declare_create_object_internal+0xb6/0x2f0 [mdd] mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd] mdd_declare_create+0x5a/0x680 [mdd] mdd_create+0x6d7/0x2550 [mdd] mdt_reint_open+0x345c/0x3ce0 [mdt] mdt_reint_rec+0x144/0x2c0 [mdt] mdt_reint_internal+0x6ad/0xdd0 [mdt] mdt_intent_open+0x187/0x5c0 [mdt] mdt_intent_opc.constprop.48+0x153/0xfa0 [mdt] ? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt] mdt_intent_policy+0x151/0x680 [mdt] ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc] ? cfs_hash_rw_unlock+0x15/0x40 [obdclass] ? cfs_hash_add+0xf6/0x2a0 [obdclass] ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc] tgt_enqueue+0xd7/0x320 [ptlrpc] tgt_handle_request0+0x139/0xb00 [ptlrpc] tgt_request_handle+0x59b/0x2010 [ptlrpc] ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc] ptlrpc_main+0xcfc/0x1410 [ptlrpc] ? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc] kthread+0x1de/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 ---[ end trace 90eb496e7f790718 ]--- Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 1054367:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1791984275/real 1791984275] req@ffff995de608aa00 x1879030194739456/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1791984291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 1054367:0:(client.c:2514:ptlrpc_expire_one_request()) Skipped 72 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1506 to 0x240000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1486 to 0x300000400:1505) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1493 to 0x2c0000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1502 to 0x280000400:1537) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1548 to 0x280000400:1569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1547 to 0x240000400:1569) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1515 to 0x300000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1547 to 0x2c0000400:1569) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 1083043:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000a491bb00 ns: mdt-lustre-MDT0000_UUID lock: 00000000c3625d9d/0x8080361a225796ff lrc: 5/0,0 mode: CW/CW res: [0x20000afe1:0x150:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x8080361a225796f1 expref: 4 pid: 1083043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-14 13:25) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 3439466:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3439466:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 3439466 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x119c/0x1c70 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fa73f78f8fb | Lustre: DEBUG MARKER: Started rundbench load pid=3434600 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3929 to 0x280000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3885 to 0x300000400:3905) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3885 to 0x2c0000400:3905) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3885 to 0x240000400:3905) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 3376911:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff942288df4600 x1878664728929664/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3933 to 0x240000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3933 to 0x300000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3997 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3933 to 0x2c0000400:3969) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4059 to 0x280000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3995 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3994 to 0x2c0000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3995 to 0x300000400:4033) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4053 to 0x300000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4117 to 0x280000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4053 to 0x240000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4053 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff94220ce26000: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 36 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4123 to 0x2c0000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4123 to 0x240000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4188 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4124 to 0x300000400:4161) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 3437425:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000220483bd ns: mdt-lustre-MDT0000_UUID lock: 00000000d32e1768/0x3a6299e2e08caa0b lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x111a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x3a6299e2e08ca9fd expref: 3 pid: 3437425 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 41 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4189 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4190 to 0x300000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4253 to 0x280000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4190 to 0x2c0000400:4225) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4317 to 0x280000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4253 to 0x2c0000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4253 to 0x240000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4252 to 0x300000400:4289) Lustre: 3376905:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1791636953/real 1791636953] req@ffff942287237000 x1878664732336000/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1791636969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 3376905:0:(client.c:2514:ptlrpc_expire_one_request()) Skipped 64 previous similar messages Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4371 to 0x280000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4307 to 0x2c0000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4307 to 0x300000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4307 to 0x240000400:4353) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 3439083:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000007c735b66 ns: mdt-lustre-MDT0000_UUID lock: 0000000096d251af/0x3a6299e2e08f035c lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1227:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x3a6299e2e08f034e expref: 4 pid: 3439083 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-10 12:56) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 3641513:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3641513:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 3641513 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? ldlm_resource_free+0x110/0x240 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x56/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x119c/0x1c70 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f78cd2878fb | Lustre: DEBUG MARKER: Started rundbench load pid=3641330 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff9c4d11cce000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages LustreError: 3638669:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000003e028197 ns: mdt-lustre-MDT0000_UUID lock: 0000000032f945ef/0x16cfd4b829185cb5 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf15:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x16cfd4b829185ca7 expref: 2 pid: 3638669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-10 11:43) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 3327671:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3327671:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 3327671 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? native_safe_halt+0x32/0x50 cleanup_resource+0x30d/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x119c/0x1c70 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f336e63a8fb | Lustre: DEBUG MARKER: Started rundbench load pid=3324381 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3959 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3914 to 0x240000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 28 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3956 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3957 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4020 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3956 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 3326067:0:(osd_handler.c:724:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 3326067:0:(osd_handler.c:724:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffffa08f635ad800: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000-mdc-ffffa08f635ad800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages LustreError: 3325711:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000009471687b ns: mdt-lustre-MDT0000_UUID lock: 0000000043a1e56b/0xdf81f582befc4296 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf04:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xdf81f582befc4288 expref: 4 pid: 3325711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4021 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4021 to 0x2c0000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4021 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4085 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4155 to 0x280000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4092 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4091 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 3327262:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000f0ee2e4a ns: mdt-lustre-MDT0000_UUID lock: 000000000b46331f/0xdf81f582befd8362 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xee7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xdf81f582befd8354 expref: 3 pid: 3327262 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0 | Link to test (2026-10-02 15:21) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1058812:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1058812:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 1058812 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? ldlm_resource_free+0x110/0x240 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x41c/0xea0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f30175dc8fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff9d81845d1000: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1229 to 0x300000400:1249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1230 to 0x280000400:1249) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1229 to 0x2c0000400:1249) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1231 to 0x240000400:1249) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1307 to 0x240000400:1345) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1308 to 0x280000400:1345) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1308 to 0x2c0000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1308 to 0x300000400:1345) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1407 to 0x300000400:1441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1407 to 0x240000400:1441) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1406 to 0x2c0000400:1441) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1406 to 0x280000400:1441) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 1032510:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790651861/real 1790651861] req@ffff9d8216c2f000 x1877633084922368/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790651877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 1032510:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 62 previous similar messages LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1491 to 0x2c0000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1490 to 0x280000400:1505) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1491 to 0x240000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1491 to 0x300000400:1537) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 1032506:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9d820fe6e200 x1877633086103936/t111669150235(111669150235) o101->lustre-MDT0000-mdc-ffff9d81c52c5000@0@lo:12/10 lens 576/608 e 0 to 0 dl 1790651949 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 1032506:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 406 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1549 to 0x240000400:1569) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1517 to 0x280000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1549 to 0x300000400:1569) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1549 to 0x2c0000400:1569) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff9d81845d1000: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0002: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 55 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1601) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1580 to 0x240000400:1601) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1549 to 0x280000400:1569) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1581 to 0x300000400:1601) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 1032521:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d8188f5bb80 x1877633088155008/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 63 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1648 to 0x300000400:1665) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1647 to 0x240000400:1665) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1615 to 0x280000400:1633) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1647 to 0x2c0000400:1665) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.33@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1721 to 0x240000400:1761) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1688 to 0x280000400:1729) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1721 to 0x2c0000400:1761) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1720 to 0x300000400:1761) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 1032520:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d81f0e54d00 x1877633089817472/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1032520:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 5 previous similar messages LustreError: 1057888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 1057888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1782 to 0x280000400:1825) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1814 to 0x300000400:1857) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1814 to 0x2c0000400:1857) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1813 to 0x240000400:1857) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 1032511:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d81ec963100 x1877633090835584/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1032511:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: 1058767:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000090588ecf ns: mdt-lustre-MDT0000_UUID lock: ffff9d8233b75600/0xeb330eaf92cf144d lrc: 4/0,0 mode: PR/PR res: [0x20000afe1:0x20b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xeb330eaf92cf1431 expref: 3 pid: 1058767 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-29 03:21) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 1126805:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1126518:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000ad0682ce ns: mdt-lustre-MDT0000_UUID lock: ffff9137cb35fc00/0x6e5c64ab4aa95343 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xee0:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x6e5c64ab4aa95335 expref: 4 pid: 1126518 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 1126805:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 1126805 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x41c/0xea0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f51359518fb | Lustre: DEBUG MARKER: Started rundbench load pid=1120752 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3918 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3962 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3949 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3918 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3963 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3996 to 0x300000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4028 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3963 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff913794c55000: operation mds_close to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-mdc-ffff913794c55000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0002: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4025 to 0x280000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4025 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4089 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4056 to 0x300000400:4097) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4087 to 0x2c0000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4151 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4118 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4087 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4157 to 0x280000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4156 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4189 to 0x300000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4220 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.93@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4284 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4252 to 0x300000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4221 to 0x280000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4220 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 1063061:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9137bd2b2300 x1877418430070656/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1063061:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 1063052:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790448420/real 1790448420] req@ffff9137f950c600 x1877418429372288/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790448436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 1063052:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 67 previous similar messages LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4350 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4286 to 0x2c0000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4317 to 0x280000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4317 to 0x300000400:4353) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4373 to 0x300000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4404 to 0x240000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4340 to 0x2c0000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4372 to 0x280000400:4417) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 1120777:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff913794c55000: can't stat MDS #0: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 1120777:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 1 previous similar message LustreError: 1063052:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9137faee5b00 x1877418431468672/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1063052:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 1063047:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff913747218000 x1877418425730944/t317827580344(317827580344) o101->lustre-MDT0000-mdc-ffff913794c55000@0@lo:12/10 lens 576/608 e 0 to 0 dl 1790448522 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 1063047:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 449 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4440 to 0x300000400:4481) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4472 to 0x240000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4408 to 0x2c0000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4440 to 0x280000400:4481) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 1125826:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000677d0955 ns: mdt-lustre-MDT0000_UUID lock: ffff913746ea3600/0x6e5c64ab4aa8926f lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xefe:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x6e5c64ab4aa89261 expref: 4 pid: 1125826 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4542 to 0x240000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4479 to 0x2c0000400:4513) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4510 to 0x280000400:4545) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4510 to 0x300000400:4545) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 | Link to test (2026-09-26 18:49) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 3681033:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3681033:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 3681033 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] ldlm_resource_clean+0x56/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] LustreError: 3670889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789965041 with bad export cookie 10855493782215354784 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x41c/0xea0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fc18f3b18fb | Lustre: DEBUG MARKER: Started rundbench load pid=3678506 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3953 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 3678892:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000add32713 ns: mdt-lustre-MDT0000_UUID lock: ffff8cffd805d600/0x96a676147e6705b1 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x96a676147e6705a3 expref: 4 pid: 3678892 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3999 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4091 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4058 to 0x2c0000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4027 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4026 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8cff8db97000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 6 previous similar messages LustreError: 3678532:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffff8cff8db97000: can't stat MDS #0: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 3678532:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 1 previous similar message Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-mdc-ffff8cff8db97000: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 37 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4115 to 0x2c0000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4083 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4083 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4147 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 3680649:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000f81ec13f ns: mdt-lustre-MDT0000_UUID lock: ffff8d0056ef5800/0x96a676147e69202c lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x10ab:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x96a676147e69201e expref: 3 pid: 3680649 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-21 04:30) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1438965:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1438965:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 1438965 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f0a6bffb8fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffffa0f40f00f800: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1268 to 0x280000400:1313) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1203 to 0x300000400:1249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1266 to 0x240000400:1281) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1267 to 0x2c0000400:1313) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 28 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1362 to 0x280000400:1377) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1329 to 0x240000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1297 to 0x300000400:1313) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1361 to 0x2c0000400:1377) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 1410826:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0f411340380 x1876504457134592/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1432475:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000084290ebd ns: mdt-lustre-MDT0000_UUID lock: ffffa0f40d2ffc00/0x5dcf245af7005a83 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x48:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x5dcf245af7005a75 expref: 4 pid: 1432475 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1432 to 0x280000400:1473) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1369 to 0x300000400:1409) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1433 to 0x2c0000400:1473) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1401 to 0x240000400:1441) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1500 to 0x240000400:1537) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1531 to 0x280000400:1569) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1532 to 0x2c0000400:1569) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1468 to 0x300000400:1505) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 1410828:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa0f49ea4a680 x1876504458670592/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 1410828:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 1410824:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffffa0f445025400 x1876504458346368/t111669150196(111669150196) o101->lustre-MDT0000-mdc-ffffa0f40f00f800@0@lo:12/10 lens 648/608 e 0 to 0 dl 1789575590 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 LustreError: 1410824:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 340 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1581 to 0x280000400:1601) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1516 to 0x300000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1548 to 0x240000400:1569) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1601) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 1433833:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0000-mdc-ffffa0f416425000: can't stat MDS #0: rc = -107 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1578 to 0x240000400:1601) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1546 to 0x300000400:1569) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1609 to 0x2c0000400:1633) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1610 to 0x280000400:1633) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 60 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1664 to 0x280000400:1697) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1632 to 0x240000400:1665) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1663 to 0x2c0000400:1697) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1600 to 0x300000400:1633) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1772 to 0x280000400:1793) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1740 to 0x240000400:1761) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1707 to 0x300000400:1729) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1772 to 0x2c0000400:1793) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1805 to 0x240000400:1825) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1773 to 0x300000400:1793) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1838 to 0x2c0000400:1857) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1838 to 0x280000400:1857) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1858 to 0x300000400:1889) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1922 to 0x2c0000400:1953) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1891 to 0x240000400:1921) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1923 to 0x280000400:1953) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1967 to 0x2c0000400:1985) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1903 to 0x300000400:1921) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1966 to 0x280000400:1985) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1935 to 0x240000400:1953) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 12 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2002 to 0x2c0000400:2017) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1938 to 0x300000400:1953) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1969 to 0x240000400:1985) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2001 to 0x280000400:2017) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 13 times Lustre: Failing over lustre-MDT0000 LustreError: 1438564:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000d178fd1e ns: mdt-lustre-MDT0000_UUID lock: ffffa0f4995dc200/0x5dcf245af70a27d9 lrc: 4/0,0 mode: PR/PR res: [0x20000afe1:0x2b5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5dcf245af70a2793 expref: 3 pid: 1438564 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-16 16:23) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 2043768:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2043768:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 13 PID: 2043768 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7ff85b77c8fb | Lustre: DEBUG MARKER: Started rundbench load pid=2043571 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 2040890:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000ab982505 ns: mdt-lustre-MDT0000_UUID lock: ffff9e83a8b80a00/0xea6ee39dfabccbb6 lrc: 5/0,0 mode: CW/CW res: [0x20001a9e3:0xf1b:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xea6ee39dfabccba8 expref: 4 pid: 2040890 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-16 11:27) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 369502:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 369115:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000c2fab3cb ns: mdt-lustre-MDT0000_UUID lock: ffff945e901b1400/0xa928cf9efdcb644f lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xef0:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xa928cf9efdcb6441 expref: 4 pid: 369115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 369502:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 13 PID: 369502 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? do_raw_spin_unlock+0x79/0x190 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f81f1a078fb | Lustre: DEBUG MARKER: Started rundbench load pid=366381 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3952 to 0x2c0000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3968 to 0x300000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4000 to 0x2c0000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4065 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 312029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832888 with bad export cookie 12189220673247782516 LustreError: 308669:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff945e7507cd00 x1875724544930304/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 308669:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 367464:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000006a2db345 ns: mdt-lustre-MDT0000_UUID lock: ffff945e1df7e000/0xa928cf9efdc8fa5e lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x101b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xa928cf9efdc8fa50 expref: 3 pid: 367464 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4030 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4127 to 0x240000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4030 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4062 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4093 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4092 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4188 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4124 to 0x2c0000400:4161) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff945dd61f6800: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 39 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4243 to 0x240000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4147 to 0x300000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4179 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4147 to 0x280000400:4193) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 | Link to test (2026-09-08 02:02) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 835107:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 835107:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 835107 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fc1f96088fb | Lustre: DEBUG MARKER: Started rundbench load pid=828487 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 820865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792326 with bad export cookie 11202927435219957756 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 826272:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000ace7ea11 ns: mdt-lustre-MDT0000_UUID lock: ffff9d6c865c5600/0x9b78cb258cc43f36 lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xef7:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x9b78cb258cc43f28 expref: 3 pid: 826272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 820870:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d6ce045d080 x1875682011778048/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 LustreError: lustre-MDT0000-mdc-ffff9d6ce0164800: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3965 to 0x2c0000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3920 to 0x240000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 773908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792352 with bad export cookie 11202927435220008947 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3964 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3964 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4029 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 770828:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d6ca9c76580 x1875682013095552/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4030 to 0x240000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4094 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4030 to 0x300000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4030 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff9d6ce0164800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4147 to 0x2c0000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4083 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4082 to 0x240000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4082 to 0x300000400:4097) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4216 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4153 to 0x280000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4121 to 0x240000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4120 to 0x300000400:4161) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 831212:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000477468c8 ns: mdt-lustre-MDT0000_UUID lock: ffff9d6ca62caa00/0x9b78cb258cc7efa3 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1119:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x9b78cb258cc7ef95 expref: 3 pid: 831212 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4221 to 0x280000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4284 to 0x2c0000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4189 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4189 to 0x300000400:4225) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 770820:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792470/real 1788792470] req@ffff9d6d10adc280 x1875682014976000/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 770820:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 58 previous similar messages LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4344 to 0x2c0000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4279 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4248 to 0x240000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4248 to 0x300000400:4289) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4338 to 0x280000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4307 to 0x300000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4306 to 0x240000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4402 to 0x2c0000400:4417) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 770819:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9d6cd8151f80 x1875682011319680/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff9d6ce0164800@0@lo:12/10 lens 576/608 e 0 to 0 dl 1788792571 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 770819:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 378 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4346 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4379 to 0x280000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4378 to 0x300000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4442 to 0x2c0000400:4481) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4505 to 0x2c0000400:4545) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4441 to 0x280000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4410 to 0x240000400:4449) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4442 to 0x300000400:4481) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4565 to 0x2c0000400:4609) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4501 to 0x280000400:4545) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4500 to 0x300000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4468 to 0x240000400:4513) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: 834717:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000023104b67 ns: mdt-lustre-MDT0000_UUID lock: ffff9d6c85f77600/0x9b78cb258ccc22b6 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x9b78cb258ccc22a8 expref: 2 pid: 834717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-07 14:50) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 802750:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 802750:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 6 PID: 802750 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fd7352968fb | Lustre: DEBUG MARKER: Started rundbench load pid=797298 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 789677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791187 with bad export cookie 10041485578729037495 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3918 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3918 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3981 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3994 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3964 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4060 to 0x240000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3964 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4028 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff9a6f20d4a800: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-mdc-ffff9a6f20d4a800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4120 to 0x240000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4025 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4024 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4088 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4184 to 0x240000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4087 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4088 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4152 to 0x300000400:4193) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4220 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4156 to 0x2c0000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4251 to 0x240000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4156 to 0x280000400:4193) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.3@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4319 to 0x240000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4223 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4286 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4222 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: 739619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788791322/real 1788791322] req@ffff9a6ea7acbf00 x1875680824591104/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788791338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 739619:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 70 previous similar messages Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4376 to 0x240000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4344 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4279 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4280 to 0x2c0000400:4321) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4435 to 0x240000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4402 to 0x300000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4339 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4338 to 0x2c0000400:4353) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 739610:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff9a6ecbf49f80 x1875680820855680/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff9a6f20d4a800@0@lo:12/10 lens 576/608 e 0 to 0 dl 1788791424 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 739610:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 417 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4508 to 0x240000400:4545) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4412 to 0x280000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4379 to 0x2c0000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4443 to 0x300000400:4481) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 802359:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000fcda9e12 ns: mdt-lustre-MDT0000_UUID lock: ffff9a6ed50a2600/0x8b5a85ee89618e16 lrc: 5/0,0 mode: CW/CW res: [0x20001a9e3:0x128a:0x0].0x0 bits 0x5/0x0 rrc: 5 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x8b5a85ee89618e08 expref: 4 pid: 802359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-07 14:30) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 150553:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 150553:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 150553 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f69f229f8fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8aa24593d000: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1233 to 0x2c0000400:1249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1233 to 0x280000400:1249) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1234 to 0x240000400:1249) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1233 to 0x300000400:1249) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1320 to 0x240000400:1345) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1320 to 0x300000400:1345) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1321 to 0x280000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1320 to 0x2c0000400:1345) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 140116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 140116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1434 to 0x300000400:1473) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1434 to 0x2c0000400:1473) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1434 to 0x280000400:1473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1434 to 0x240000400:1473) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1495 to 0x240000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1495 to 0x2c0000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1496 to 0x300000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1495 to 0x280000400:1537) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 140756:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000077766c15 ns: mdt-lustre-MDT0000_UUID lock: ffff8aa22ae38600/0x571985fad36946e1 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x3d:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x571985fad36946a9 expref: 3 pid: 140756 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 117999:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8aa26cd72680 x1875436216001024/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 117985:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8aa229f69f80 x1875436211085312/t94489280696(94489280696) o101->lustre-MDT0000-mdc-ffff8aa25b1da000@0@lo:12/10 lens 648/608 e 0 to 0 dl 1788556782 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 117985:0:(client.c:3449:ptlrpc_replay_interpret()) Skipped 470 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1555 to 0x240000400:1601) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1555 to 0x280000400:1601) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1555 to 0x300000400:1601) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1556 to 0x2c0000400:1601) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 141539:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000069898d02 ns: mdt-lustre-MDT0000_UUID lock: ffff8aa25aa6c000/0x571985fad36a2d4e lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x1b4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x571985fad36a2d39 expref: 3 pid: 141539 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 50 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1663 to 0x2c0000400:1697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1664 to 0x240000400:1697) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1664 to 0x300000400:1697) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1663 to 0x280000400:1697) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 11 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 60 previous similar messages Lustre: 117999:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788556817/real 1788556817] req@ffff8aa26cd7b100 x1875436218017280/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788556833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 117999:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 74 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1770 to 0x300000400:1793) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1771 to 0x2c0000400:1793) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1770 to 0x280000400:1793) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1770 to 0x240000400:1793) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.73@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1864 to 0x2c0000400:1889) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1865 to 0x240000400:1889) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1865 to 0x280000400:1889) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1865 to 0x300000400:1889) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 143866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 143866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1901 to 0x240000400:1921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1901 to 0x280000400:1921) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1902 to 0x300000400:1921) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1901 to 0x2c0000400:1921) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8aa24593d000: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 17 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1940 to 0x2c0000400:1985) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1940 to 0x300000400:1985) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1940 to 0x280000400:1985) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1939 to 0x240000400:1985) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2045 to 0x240000400:2081) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2045 to 0x300000400:2081) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2045 to 0x280000400:2081) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2045 to 0x2c0000400:2081) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 12 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2130 to 0x2c0000400:2145) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2130 to 0x240000400:2177) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2130 to 0x280000400:2145) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2130 to 0x300000400:2177) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 13 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2214 to 0x280000400:2241) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2246 to 0x240000400:2273) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2214 to 0x2c0000400:2241) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2247 to 0x300000400:2273) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 14 times Lustre: Failing over lustre-MDT0000 LustreError: 117990:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8aa25acf1500 x1875436224545280/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 LustreError: 117990:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2310 to 0x240000400:2337) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2310 to 0x300000400:2337) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2278 to 0x2c0000400:2305) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2278 to 0x280000400:2305) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 15 times Lustre: Failing over lustre-MDT0000 LustreError: 147038:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000bd05cda4 ns: mdt-lustre-MDT0000_UUID lock: ffff8aa26b214400/0x571985fad3745a04 lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x2e0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x571985fad37459da expref: 3 pid: 147038 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 117998:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8aa1d0683800 x1875436225358208/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 LustreError: 117998:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2359 to 0x240000400:2401) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2327 to 0x280000400:2369) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2360 to 0x300000400:2401) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2327 to 0x2c0000400:2369) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 16 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2400 to 0x280000400:2433) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2432 to 0x300000400:2465) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2432 to 0x240000400:2465) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2400 to 0x2c0000400:2433) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 17 times Lustre: Failing over lustre-MDT0000 LustreError: 148381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 148381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2534 to 0x240000400:2561) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2534 to 0x300000400:2561) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2502 to 0x280000400:2529) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2502 to 0x2c0000400:2529) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 18 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2628 to 0x300000400:2657) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2595 to 0x2c0000400:2625) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2628 to 0x240000400:2657) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2596 to 0x280000400:2625) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 19 times Lustre: Failing over lustre-MDT0000 LustreError: 149840:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0xf6a:0x0].0x0 (ffff8aa258fccc00) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2680 to 0x2c0000400:2721) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2681 to 0x280000400:2721) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2712 to 0x240000400:2753) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2712 to 0x300000400:2753) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 20 times Lustre: Failing over lustre-MDT0000 LustreError: 140550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788557178 with bad export cookie 6276194868053733458 LustreError: 117990:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8aa2789cad80 x1875436230295424/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 LustreError: 117990:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 17 previous similar messages LustreError: 150038:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000b3cfd021 ns: mdt-lustre-MDT0000_UUID lock: ffff8aa1ce72d400/0x571985fad37a4ca7 lrc: 4/0,0 mode: CW/CW res: [0x20000afe1:0x54d:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x571985fad37a4c84 expref: 4 pid: 150038 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-09-04 21:26) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 343441:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 343441:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 343441 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fb8eb3208fb | Lustre: DEBUG MARKER: Started rundbench load pid=342671 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 15 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3963 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3918 to 0x2c0000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3918 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3919 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 343053:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000000c275c8d ns: mdt-lustre-MDT0000_UUID lock: ffff8a6c9d55be00/0xcca06d024d43994a lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf92:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xcca06d024d43993c expref: 3 pid: 343053 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-31 23:07) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 81899:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 81899:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 13 PID: 81899 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xac0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7efd38a698fb | Lustre: DEBUG MARKER: Started rundbench load pid=75501 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 19 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3917 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3948 to 0x280000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3917 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3991 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3960 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3955 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4017 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff92b40f98f000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 33 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.53@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4052 to 0x280000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4022 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4043 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 77969:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 77969:0:(osd_handler.c:720:osd_ro()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff92b40f98f000: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages LustreError: 3285:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff92b46cacad80 x1875007542182528/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 LustreError: 3285:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4090 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4089 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4122 to 0x280000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4084 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4143 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4145 to 0x300000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4176 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4140 to 0x240000400:4161) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4187 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4211 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4186 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4183 to 0x240000400:4225) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 80048:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000d243834a ns: mdt-lustre-MDT0000_UUID lock: ffff92b469b44800/0x97923223b6695d6a lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xee5:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x97923223b6695d5c expref: 4 pid: 80048 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788149267/real 1788149267] req@ffff92b4625b3800 x1875007543195520/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788149283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 3287:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4273 to 0x280000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4246 to 0x300000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4244 to 0x2c0000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4235 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4309 to 0x280000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4309 to 0x300000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4310 to 0x2c0000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4280 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 81472:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000003993d3a7 ns: mdt-lustre-MDT0000_UUID lock: ffff92b456351a00/0x97923223b66ab209 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x118f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x97923223b66ab1fb expref: 4 pid: 81472 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-31 04:09) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 2500630:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2500630:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 2500630 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f6e8db1e8fb | Lustre: DEBUG MARKER: Started rundbench load pid=2498683 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3962 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3917 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3918 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3918 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 2499173:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000009b73f03f ns: mdt-lustre-MDT0000_UUID lock: ffffa0f0cec90000/0x92d6b8ecb471119 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xedd:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x92d6b8ecb47110b expref: 4 pid: 2499173 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3963 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3963 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3963 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4027 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 33 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4020 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4084 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4020 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffffa0f0b4140800: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages LustreError: 2500244:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000d0a970ec ns: mdt-lustre-MDT0000_UUID lock: ffffa0f0e1142c00/0x92d6b8ecb489c4c lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1034:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x92d6b8ecb489c3e expref: 4 pid: 2500244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-30 02:56) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 3565920:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3565630:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000083b7f334 ns: mdt-lustre-MDT0000_UUID lock: ffff932dc524dc00/0x48013b3dc4ca58f4 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1207:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x48013b3dc4ca58e6 expref: 4 pid: 3565630 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 3565920:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 3565920 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? do_raw_spin_unlock+0x79/0x190 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f16965068fb | Lustre: DEBUG MARKER: Started rundbench load pid=3558760 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 3486088:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff932d83269180 x1874890953587072/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3915 to 0x280000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3915 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4053 to 0x240000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3957 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3956 to 0x2c0000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3956 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 33 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4020 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4117 to 0x240000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4020 to 0x300000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4020 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 3561206:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 3561206:0:(osd_handler.c:720:osd_ro()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4185 to 0x240000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4089 to 0x2c0000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4090 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4089 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff932dbd532800: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4242 to 0x240000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4145 to 0x300000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4145 to 0x2c0000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4145 to 0x280000400:4161) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 3563205:0:(ldlm_lib.c:2976:target_recovery_thread()) too long recovery - read logs LustreError: dumping log to /tmp/lustre-log.1788038137.3563205 Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4184 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4184 to 0x280000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4184 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4280 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.93@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4346 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4250 to 0x2c0000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4251 to 0x280000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4250 to 0x300000400:4289) Lustre: 3486091:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788038144/real 1788038144] req@ffff932d7ae5f380 x1874890956821760/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788038160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 3486091:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 70 previous similar messages Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 3486085:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff932db8f0f380 x1874890953362432/t317827580346(317827580346) o101->lustre-MDT0000-mdc-ffff932dbd532800@0@lo:12/10 lens 576/608 e 0 to 0 dl 1788038201 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3486085:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 326 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4304 to 0x280000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4401 to 0x240000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4305 to 0x2c0000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4305 to 0x300000400:4321) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4345 to 0x2c0000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4345 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4441 to 0x240000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4345 to 0x300000400:4385) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 | Link to test (2026-08-29 21:17) |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | LustreError: 1662531:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1662531:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 1662531 Comm: umount Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f046f9818fb | Lustre: DEBUG MARKER: Started rundbench load pid=1661209 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): 6 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.33@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3748 to 0x2c0000402:3777) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3745) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3745) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3745) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-mdc-ffff8b6318a4f000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8b6318a4f000: can't stat MDS #0: rc = -107 LustreError: 1605152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 1605152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 119 previous similar messages LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8b6318a4f000: can't stat MDS #0: rc = -19 LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8b6318a4f000: can't stat MDS #0: rc = -19 LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8b6318a4f000: can't stat MDS #0: rc = -19 LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) lustre-MDT0001-mdc-ffff8b6318a4f000: can't stat MDS #0: rc = -19 LustreError: 1661234:0:(lmv_obd.c:1471:lmv_statfs()) Skipped 1 previous similar message LDISKFS-fs (dm-1): 12 truncates cleaned up LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:225) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:225) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:225) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:203 to 0x340000401:225) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 LustreError: 1606211:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000028bb2444 ns: mdt-lustre-MDT0002_UUID lock: ffff8b630263b800/0x5fb42379a3449023 lrc: 4/0,0 mode: PR/PR res: [0x2800007ed:0x3bb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5fb42379a3449015 expref: 3 pid: 1606211 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-29 09:47) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 481260:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 481260:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 4 PID: 481260 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7fb74a4d98fb | Lustre: DEBUG MARKER: Started rundbench load pid=480279 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 477917:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000002d8d751a ns: mdt-lustre-MDT0000_UUID lock: ffff95d21ba85c00/0x24ce7d8cc33bcdeb lrc: 5/0,0 mode: PR/PR res: [0x20001a9e3:0xf1c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x24ce7d8cc33bcddd expref: 3 pid: 477917 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3991 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3947 to 0x2c0000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3947 to 0x280000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3946 to 0x300000400:3969) Lustre: DEBUG MARKER: centos-86.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 480833:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000007c940f68 ns: mdt-lustre-MDT0000_UUID lock: ffff95d21a168200/0x24ce7d8cc33c59b3 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf59:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x24ce7d8cc33c59a5 expref: 3 pid: 480833 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-28 12:51) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 313960:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 313536:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000009197c908 ns: mdt-lustre-MDT0000_UUID lock: ffff9eaa3a796000/0x57943875b988f945 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x10b6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x57943875b988f937 expref: 3 pid: 313536 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: 313960:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 8 PID: 313960 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f74cf1f28fb | Lustre: DEBUG MARKER: Started rundbench load pid=309922 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff9eaaacebb000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages LustreError: 307851:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000080e9742c ns: mdt-lustre-MDT0000_UUID lock: ffff9eaa0a59a200/0x57943875b985aba3 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xf1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x57943875b985ab95 expref: 3 pid: 307851 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.98@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 32 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3915 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3954 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4020 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3959 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3956 to 0x300000400:4001) Lustre: 237238:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787701251/real 1787701251] req@ffff9eaa8cb81880 x1874537772435328/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787701267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 237238:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 311595:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 311595:0:(osd_handler.c:715:osd_ro()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4086 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3988 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4026 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4155 to 0x240000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4058 to 0x280000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4091 to 0x300000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff9eaaacebb000: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4146 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4153 to 0x300000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4126 to 0x280000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4215 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 | Link to test (2026-08-25 23:42) |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 3064482:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3064482:0:(ldlm_lockd.c:1018:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 3064482 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x2e6/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f9345f9e8fb | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 3034485:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff99e018ed7700 x1874479966450688/t94489280716(94489280716) o101->lustre-MDT0000-mdc-ffff99e024cae800@0@lo:12/10 lens 648/608 e 0 to 0 dl 1787644808 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3034485:0:(client.c:3448:ptlrpc_replay_interpret()) Skipped 2 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1349 to 0x280000400:1377) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1318 to 0x240000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1222 to 0x2c0000400:1249) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:998 to 0x300000400:1025) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 3061632:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000f97417bf ns: mdt-lustre-MDT0000_UUID lock: ffff99e06e802200/0x54845280a64aa413 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x3a:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x54845280a64aa405 expref: 4 pid: 3061632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 32 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1301 to 0x2c0000400:1345) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1403 to 0x240000400:1441) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1078 to 0x300000400:1121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1421 to 0x280000400:1441) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 3034489:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787644853/real 1787644853] req@ffff99e01693d080 x1874479968313984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787644869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: 3034489:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 57 previous similar messages Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1395 to 0x2c0000400:1441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1503 to 0x240000400:1537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1489 to 0x280000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1165 to 0x300000400:1185) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 3063168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1597 to 0x240000400:1633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1588 to 0x280000400:1633) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1496 to 0x2c0000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1250 to 0x300000400:1281) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 3064109:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000e2940198 ns: mdt-lustre-MDT0000_UUID lock: ffff99e007941c00/0x54845280a64d65a7 lrc: 4/0,0 mode: CW/CW res: [0x20000afe1:0x11c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x54845280a64d6599 expref: 4 pid: 3064109 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-25 08:01) |
| replay-single test 70b: dbench 1mdts recovery; 1 clients | LustreError: 109802:0:(ldlm_lockd.c:1058:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 109802:0:(ldlm_lockd.c:1058:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 109802 Comm: umount Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ldlm_server_completion_ast+0x635/0xd80 [ptlrpc] ? string+0x60/0x80 cleanup_resource+0x30d/0x490 [ptlrpc] ldlm_resource_clean+0x3f/0x70 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_relax+0x267/0x5f0 [obdclass] ? cleanup_resource+0x490/0x490 [ptlrpc] ? cleanup_resource+0x490/0x490 [ptlrpc] cfs_hash_for_each_nolock+0x1a0/0x2c0 [obdclass] ldlm_namespace_cleanup+0x38/0xf0 [ptlrpc] __ldlm_namespace_free+0x72/0x6a0 [ptlrpc] ? _cond_resched+0x21/0x50 ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 ldlm_namespace_free_prior+0x87/0x2c0 [ptlrpc] mdt_device_fini+0x233/0xb70 [mdt] obd_precleanup.isra.17+0xb3/0x380 [obdclass] ? class_disconnect_exports+0x1d1/0x3b0 [obdclass] class_cleanup+0x410/0xab0 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? _raw_spin_unlock+0x16/0x30 class_process_config+0xdb0/0x2450 [obdclass] class_manual_cleanup+0x4b0/0xa60 [obdclass] ? do_raw_spin_unlock+0x79/0x190 ? class_name2obd+0x142/0x190 [obdclass] server_put_super+0x11dc/0x1cb0 [ptlrpc] ? fsnotify_grab_connector+0x5b/0xb0 ? fsnotify_sb_delete+0x234/0x320 generic_shutdown_super+0xb7/0x1c0 kill_anon_super+0x20/0x50 lustre_kill_super+0x2e/0x60 [lustre] deactivate_locked_super+0x59/0xd0 deactivate_super+0x88/0xa0 cleanup_mnt+0x63/0xe0 __cleanup_mnt+0x1a/0x30 task_work_run+0xd2/0x120 exit_to_usermode_loop+0x1e4/0x200 do_syscall_64+0x3de/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f6997fa28fb | Lustre: DEBUG MARKER: Started rundbench load pid=109598 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 107259:0:(ldlm_lockd.c:1509:ldlm_handle_enqueue()) ### lock on destroyed export 00000000c22ccbde ns: mdt-lustre-MDT0000_UUID lock: ffff9e1e2941b400/0x85192323b794a97d lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf24:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x85192323b794a96f expref: 4 pid: 107259 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 | Link to test (2026-08-22 14:22) |