| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-quota test 7a: Quota reintegration (global index) | LustreError: 53861:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 53861:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 53861, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-09-04 06:17) |
| replay-ost-single test 8b: Verify redo io: redo io should success after recovery | LustreError: 19440:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 19440:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 19440, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=230, val=0*** Lustre: Failing over lustre-OST0000 | Link to test (2026-09-03 06:51) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 13662:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 13662:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 13662, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d464754f800.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d464754f800.ost_server_uuid in DISCONN state after 0 sec Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.117@tcp (at 0@lo) Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 Lustre: DEBUG MARKER: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 Lustre: Failing over lustre-OST0001 | Link to test (2026-08-31 05:00) |
| replay-single test 47: MDS->OSC failure during precreate cleanup (2824) | LustreError: 94906:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 94906:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 94906, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-08-28 11:10) | |
| replay-single test 89: no disk space leak on late ost connection | LustreError: 205708:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 205708:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 205708, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-08-26 09:13) | |
| replay-single test 93a: replay + reconnect | LustreError: 253372:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 253372:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 253372, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-08-25 08:04) | |
| replay-ost-single test 8b: Verify redo io: redo io should success after recovery | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=230, val=0*** Lustre: Skipped 1 previous similar message Lustre: *** cfs_fail_loc=230, val=0*** Lustre: Skipped 5 previous similar messages Lustre: Failing over lustre-OST0000 | Link to test (2026-08-13 05:26) |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 37082:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37082:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37082, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=32a, val=0*** LustreError: 6317:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000994d49b1 x1873305977452672/t0(0) o105->lustre-OST0000@192.168.201.20@tcp:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' | Link to test (2026-08-12 05:22) |
| replay-ost-single test 4: Fail OST during read, with verification | LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-08-06 05:41) |
| conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows | LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 36, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local vdc: vdc1 vdc9 vdc: vdc1 vdc9 vde: vde1 vde9 vde: vde1 vde9 vde: vde1 vde9 vdf: vdf1 vdf9 Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: *** cfs_fail_loc=1e0, val=0*** Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: On OST0, 11872 inodes available. Want 100500. rc=0 LustreError: 25032:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: Failing over lustre-OST0000 | Link to test (2026-08-02 06:05) |
| conf-sanity test 47: server restart does not make client loss lru_resize settings | LustreError: 62772:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 62772:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 62772, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local vdc: vdc1 vdc9 vde: vde1 vde9 vde: vde1 vde9 vde: vde1 vde9 vdf: vdf1 vdf9 Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Failing over lustre-OST0000 | Link to test (2026-07-31 06:50) |
| replay-ost-single test 9: Verify that no req deadline happened during recovery | LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 38, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | LustreError: 27592:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: Failing over lustre-OST0000 | Link to test (2026-07-30 05:35) |
| replay-ost-single test 4: Fail OST during read, with verification | LustreError: 18299:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 18299:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 18299, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-07-30 05:09) |
| conf-sanity test 122b: Check OST sequence wouldn't change when IDIF 32bit overflows | LustreError: 21132:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 21132:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 21132, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local vdc: vdc1 vdc9 vde: vde1 vde9 vdf: vdf1 vdf9 Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all 8 Lustre: *** cfs_fail_loc=1e0, val=0*** Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: On OST0, 11872 inodes available. Want 100500. rc=0 hrtimer: interrupt took 17040788 ns LustreError: 24121:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: Failing over lustre-OST0000 | Link to test (2026-07-25 04:54) |
| replay-dual test 25: replay|resend | LustreError: 41532:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 41532:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 41532, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=304, val=0*** | Link to test (2026-07-24 06:00) |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 36, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-07-13 05:29) | |
| sanity test 27oo: don't let few threads to reserve too many objects | LustreError: 30640:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 30640:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 30640, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-07-13 04:52) |
| conf-sanity test 84: check recovery_hard_time | LustreError: 99602:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 99602:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 99602, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all 8 LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey Lustre: lustre-OST0001: new disk, initializing Lustre: Skipped 2 previous similar messages Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost Lustre: Skipped 2 previous similar messages Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: 66655:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783588207/real 1783588207] req@00000000142ced37 x1870227233150976/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783588214 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' LustreError: 166-1: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 33 previous similar messages Lustre: Evicted from MGS (at 192.168.204.142@tcp) after server handle changed from 0x1bd603beb6214a68 to 0x1bd603beb621d83d Lustre: MGC192.168.204.142@tcp: Connection restored to (at 0@lo) Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: 98645:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all 8 Lustre: 95882:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 Lustre: 95882:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 101037:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 101037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 101037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages Lustre: 101037:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout Lustre: 101037:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: lustre-MDT0000: disconnecting 1 stale clients LustreError: 101037:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 Lustre: 101037:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@0000000029de9876 x1870227223637888/t0(8589934775) o101->8b5dda53-1277-4c33-b56e-1bbd20e2edbd@192.168.204.42@tcp:458/0 lens 592/0 e 0 to 0 dl 1783588298 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:545 Lustre: lustre-OST0000: deleting orphan objects from 0x0:78 to 0x0:577 | Link to test (2026-07-09 05:15) |
| replay-single test 61c: test race mds llog sync vs llog cleanup | LustreError: 128736:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 128736:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 128736, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-07-03 06:06) | |
| recovery-small test 52: failover OST under load | LustreError: 45567:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 45567:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 45567, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-06-23 05:32) | |
| sanity test 165a: ofd access log discovery | LustreError: 63458:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 63458:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 63458, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-16 07:59) |
| sanity test 165d: ofd_access_log mask works | LustreError: 181566:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 181566:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 181566, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-15 09:02) |
| replay-ost-single test 3: Fail OST during write, with verification | LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 38, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-13 04:58) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 16700:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 16700:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 16700, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff96e24591b000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96e24591b000.ost_server_uuid in DISCONN state after 0 sec Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.115@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 Lustre: DEBUG MARKER: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 Lustre: Failing over lustre-OST0001 | Link to test (2026-06-12 04:58) |
| replay-ost-single test 1: touch | LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-09 05:12) |
| sanity test 27F: Client resend delayed layout creation with non-zero size | LustreError: 62562:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 62562:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 62562, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-09 00:12) |
| sanity-flr test 32: data should be mirrored to newly created mirror | LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 36, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-06-08 05:04) |
| replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) | LustreError: 245238:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 245238:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 245238, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-06-05 06:55) | |
| replay-ost-single test 10: conflicting PW | LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 38, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-06-05 01:21) | |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=32a, val=0*** LustreError: 58654:0:(ldlm_resource.c:1127:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x8ed:0x0:0x0].0x0 (000000001b6a8368) refcount nonzero (2) after lock cleanup; forcing cleanup. | Link to test (2026-06-04 21:56) |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 46485:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 46485:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 46485, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=32a, val=0*** LustreError: 6248:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b072c092 x1867059019774336/t0(0) o105->lustre-OST0000@192.168.202.38@tcp:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' LustreError: 99560:0:(ldlm_resource.c:1127:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x985:0x0:0x0].0x0 (00000000cb88f336) refcount nonzero (2) after lock cleanup; forcing cleanup. | Link to test (2026-06-04 07:00) |
| sanity test 27z: check SEQ/OID on the MDT and OST filesystems | LustreError: 23758:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23758:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23758, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x6751 ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x6751, count: 1 Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:154/0x9a seq:0 Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.147@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x0:156 to 0x0:193 Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x6753 ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x6753, count: 2 Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1219/0x4c3 seq:0 Lustre: Failing over lustre-OST0001 | Link to test (2026-05-28 05:47) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 15 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff988161331800.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff988161331800.ost_server_uuid in DISCONN state after 0 sec Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.123@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 Lustre: Failing over lustre-OST0001 | Link to test (2026-05-18 05:08) |
| recovery-small test 52: failover OST under load | LustreError: 23306:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23306:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23306, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 17 previous similar messages LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3731 to 0x280000400:3777 Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 258 previous similar messages LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.105@tcp (at 0@lo) Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 2 previous similar messages Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3731 to 0x280000400:3809 Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: Failing over lustre-OST0000 Lustre: Skipped 1 previous similar message | Link to test (2026-05-17 05:41) |
| recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) | LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 36, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-05-11 17:58) |
| replay-ost-single test 1: touch | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-05-11 17:29) |
| replay-single test 42: recovery after ost failure | LustreError: 58138:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 58138:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 58138, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: Failing over lustre-OST0000 Lustre: Skipped 10 previous similar messages | Link to test (2026-05-09 05:57) |
| sanity-flr test 32: data should be mirrored to newly created mirror | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-05-09 05:01) |
| sanity-quota test 80: check for EDQUOT after OST failover | LustreError: 230995:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 230995:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 230995, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=a06, val=0*** Lustre: Skipped 1 previous similar message LustreError: 213454:0:(qmt_lock.c:425:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11 Lustre: Failing over lustre-OST0001 Lustre: Skipped 1 previous similar message | Link to test (2026-05-07 06:38) |
| sanity test 165a: ofd access log discovery | LustreError: 138787:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 138787:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 138787, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-05-06 07:15) |
| recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) | LustreError: 23618:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23618:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23618, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-05-02 06:16) |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 45856:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 45856:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 45856, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=32a, val=0*** LustreError: 5491:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000081b98cc4 x1863793960880128/t0(0) o105->lustre-OST0000@192.168.203.10@tcp:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' LustreError: 58489:0:(ldlm_resource.c:1127:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x8d2:0x0:0x0].0x0 (000000007d48a167) refcount nonzero (2) after lock cleanup; forcing cleanup. | Link to test (2026-04-29 05:45) |
| conf-sanity test 47: server restart does not make client loss lru_resize settings | LustreError: 124463:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 124463:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 124463, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_hostid Lustre: DEBUG MARKER: oleg226-server.virtnet: executing load_modules_local LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all 8 LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg226-server.virtnet: executing set_default_debug -1 all 8 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] Lustre: DEBUG MARKER: oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Failing over lustre-OST0000 | Link to test (2026-04-27 05:38) |
| replay-dual test 28: lock replay should be ordered: waiting after granted | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=32a, val=0*** LustreError: 58622:0:(ldlm_resource.c:1127:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x905:0x0:0x0].0x0 (00000000932ad44e) refcount nonzero (3) after lock cleanup; forcing cleanup. LustreError: 7100:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005c2cdfb4 x1863340213719744/t0(0) o105->lustre-OST0000@192.168.204.49@tcp:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' | Link to test (2026-04-24 07:08) |
| replay-ost-single test 10: conflicting PW | LustreError: 478:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 478:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 478, comm: kworker/u8:4 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-04-24 07:05) | |
| sanity test 27oo: don't let few threads to reserve too many objects | LustreError: 41846:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 41846:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 41846, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: Skipped 2 previous similar messages | Link to test (2026-04-21 05:17) |
| recovery-small test 29b: error adding new clients doesn't cause LBUG (bug 22273) | LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 38:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 38, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-04-14 22:15) |
| sanity test 27F: Client resend delayed layout creation with non-zero size | LustreError: 56488:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 56488:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 56488, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-04-14 16:57) |
| sanity test 27F: Client resend delayed layout creation with non-zero size | LustreError: 42131:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 42131:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 42131, comm: kworker/u8:4 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-04-14 16:24) |
| sanity test 259: crash at delayed truncate | LustreError: 46920:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 46920:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 46920, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=2301, val=0*** Lustre: Failing over lustre-OST0000 | Link to test (2026-04-14 15:33) |
| recovery-small test 52: failover OST under load | LustreError: 39690:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 39690:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 39690, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: Skipped 4 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Skipped 4 previous similar messages LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3741 to 0x280000400:3777 Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000: Not available for connect from 192.168.202.35@tcp (stopping) LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 296 previous similar messages LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.135@tcp (at 0@lo) Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 16 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3741 to 0x280000400:3809 Lustre: DEBUG MARKER: oleg235-server.virtnet: executing set_default_debug -1 all 8 Lustre: DEBUG MARKER: oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: Failing over lustre-OST0000 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 192.168.202.35@tcp (stopping) Lustre: Skipped 2 previous similar messages | Link to test (2026-04-14 15:27) |
| replay-single test 93a: replay + reconnect | LustreError: 214319:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 214319:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 214319, comm: kworker/u8:4 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-04-14 13:49) | |
| replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) | LustreError: 190187:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 190187:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 190187, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-04-14 13:31) | |
| replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) | LustreError: 105461:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 105461:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 105461, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-04-14 12:46) | |
| replay-single test 88: MDS should not assign same objid to different files | LustreError: 147865:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 147865:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 147865, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 130859:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776184773 with bad export cookie 3776763178227536929 | Link to test (2026-04-14 12:43) |
| replay-single test 61c: test race mds llog sync vs llog cleanup | LustreError: 142376:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 142376:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 142376, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2026-04-14 12:33) | |
| replay-single test 42: recovery after ost failure | LustreError: 79434:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 79434:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 79434, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 | Link to test (2026-04-14 11:53) |
| replay-ost-single test 5: Fail OST during iozone | LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2026-02-15 04:53) |
| sanity test 259: crash at delayed truncate | LustreError: 23886:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23886:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23886, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: *** cfs_fail_loc=2301, val=0*** Lustre: Failing over lustre-OST0000 | Link to test (2026-01-28 05:42) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 36:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 36, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [<0>] ofd_destroy_export+0x76/0x170 [ofd] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2025-12-25 04:42) |
| sanity test 27z: check SEQ/OID on the MDT and OST filesystems | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x678c ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x678c, count: 1 Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:251/0xfb seq:0 Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 Lustre: lustre-OST0000: in recovery but waiting for the first client to connect Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.159@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x0:253 to 0x0:289 Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 Lustre: DEBUG MARKER: FID seq 0x200000401, oid 0x678e ver 0x0 Lustre: DEBUG MARKER: LOV seq 0x200000401, oid 0x678e, count: 2 Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:1187/0x4a3 seq:0 Lustre: Failing over lustre-OST0001 | Link to test (2025-11-28 05:26) |
| insanity test 10: Tenth Failure Mode: MDT0/OST/MDT1 Thu Nov 27 05:06:26 EST 2025 | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764237989/real 1764237989] req@000000003e170a54 x1849935832378048/t0(0) o400->MGC192.168.204.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764237996 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' LustreError: 166-1: MGC192.168.204.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test (2025-11-27 05:10) |
| replay-ost-single test 10: conflicting PW | LustreError: 30128:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 30128:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 30128, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2025-10-24 09:34) | |
| sanity test 165d: ofd_access_log mask works | LustreError: 150647:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 150647:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 150647, comm: kworker/u8:3 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2025-10-23 21:27) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 1362:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 1362:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 1362, comm: kworker/u8:4 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2025-10-23 17:43) |
| replay-single test 89: no disk space leak on late ost connection | LustreError: 241131:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 241131:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 241131, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: Skipped 12 previous similar messages | Link to test (2025-10-22 06:26) |
| sanity test 165b: ofd access log entries are produced and consumed | LustreError: 104091:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 104091:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 104091, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 | Link to test (2025-10-16 22:22) |
| replay-single test 93a: replay + reconnect | LustreError: 259816:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 259816:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 259816, comm: kworker/u8:1 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2025-10-16 18:19) | |
| replay-single test 135: Server failure in lock replay phase | LustreError: 83299:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 83299:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 83299, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 | Link to test (2025-10-16 15:10) |
| sanity-quota test 7b: Quota reintegration (slave index) | LustreError: 37656:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37656:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37656, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | LustreError: 78334:0:(qsd_reint.c:635:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed. LustreError: 78334:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 8 previous similar messages Lustre: *** cfs_fail_loc=a02, val=0*** LustreError: 3019:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8 qsd:lustre-OST0000 qtype:usr id:60000 enforced:1 granted: 1026 pending:0 waiting:0 req:1 usage: 2052 qunit:0 qtune:0 edquot:0 default:no Lustre: Failing over lustre-OST0000 | Link to test (2025-09-11 06:06) |
| replay-ost-single test 9: Verify that no req deadline happened during recovery | LustreError: 476:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 476:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 476, comm: kworker/u8:4 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | LustreError: 27513:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 Lustre: Failing over lustre-OST0000 | Link to test (2025-08-19 05:15) |
| sanity-flr test 33b: avoid reading from unhealthy mirror | LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8, comm: kworker/u8:0 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Lustre: Failing over lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 24 previous similar messages Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89dbe00c2800.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89dbe00c2800.ost_server_uuid in DISCONN state after 0 sec LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.107@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:27 to 0x280000400:161 Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 Lustre: DEBUG MARKER: oleg107-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 Lustre: Failing over lustre-OST0001 | Link to test (2025-08-02 04:48) |
| replay-ost-single test 10: conflicting PW | LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 37:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 37, comm: kworker/u8:2 4.18.0rh8.10-debug #2 SMP Mon Jul 14 01:24:22 EDT 2025 Call Trace TBD: [<0>] cfs_stack_trace_save_tsk+0x1b/0x30 [libcfs] [<0>] libcfs_call_trace+0x83/0xb0 [libcfs] [<0>] lbug_with_loc+0x4f/0xa0 [libcfs] [<0>] tgt_client_free+0x405/0x4a0 [ptlrpc] [<0>] ofd_destroy_export+0x76/0x170 [ofd] [<0>] class_export_destroy+0x154/0x780 [obdclass] [<0>] obd_zombie_exp_cull+0x16/0x20 [obdclass] [<0>] process_one_work+0x2c8/0x700 [<0>] worker_thread+0x296/0x6e0 [<0>] kthread+0x1d1/0x200 [<0>] ret_from_fork+0x1f/0x30 | Link to test (2025-07-14 23:20) |