| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) | LustreError: 28414:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 28414:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 28414, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 10: resending a replayed unlink | LustreError: 16591:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 16591:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 16591, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 19: resend of open request | LustreError: 12199:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12199:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12199, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 31864:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802b8877700 x1831269111926336/t77309411443(0) o101->7cf270c1-70a8-44be-89a9-f94ed0dcce6d@0@lo:661/0 lens 664/656 e 0 to 0 dl 1746435706 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client 7cf270c1-70a8-44be-89a9-f94ed0dcce6d (at 0@lo) reconnecting Lustre: 31863:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802ea2d3ec0 x1831269111926336/t77309411443(0) o101->7cf270c1-70a8-44be-89a9-f94ed0dcce6d@0@lo:668/0 lens 664/3424 e 0 to 0 dl 1746435713 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' | Link to test |
| recovery-small test 106: lightweight connection support | LustreError: 27261:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27261:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27261, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=805, val=0*** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 5: open, unlink |X| close | LustreError: 18011:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 18011:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 18011, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 16233:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 16233:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 16233, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9289bf8: operation mds_reint to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1235 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1236 to 0x0:1281 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1235 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1299 to 0x0:1345 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802af83dd28: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 28870:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a4c83240 x1831240108430656/t0(0) o2->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-2-0.0' LustreError: 19605:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x24d:0x0].0x0 (ffff8802f9aae0c0) refcount nonzero (3) after lock cleanup; forcing cleanup. LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1451 to 0x0:1473 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1388 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1387 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1387 to 0x0:1409 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802af83dd28: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 20080:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802e7122e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e5e3d680/0x4206bbbb8c9918f0 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x30:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x4206bbbb8c9918e2 expref: 4 pid: 20080 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 28861:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a9004500 x1831240107428928/t94489280661(94489280661) o101->lustre-MDT0000-mdc-ffff8800a9289bf8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1746408551 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 28861:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 390 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1568 to 0x0:1601 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1504 to 0x0:1537 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1536 to 0x0:1569 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1503 to 0x0:1537 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802af83dd28: operation mds_readpage to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1715 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1651 to 0x0:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1683 to 0x0:1729 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1652 to 0x0:1697 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9289bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1865 to 0x0:1889 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1802 to 0x0:1825 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1833 to 0x0:1857 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1801 to 0x0:1825 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9289bf8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1909 to 0x0:1953 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1910 to 0x0:1953 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1973 to 0x0:2017 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1942 to 0x0:1985 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LustreError: 22697:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802b1784138 ns: mdt-lustre-MDT0000_UUID lock: ffff880089819300/0x4206bbbb8c9f5e97 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x239:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x4206bbbb8c9f5e89 expref: 4 pid: 22697 timeout: 0 lvb_type: 0 LustreError: 11-0: lustre-MDT0000-mdc-ffff8802af83dd28: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 21763:0:(ldlm_lockd.c:2527:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746408680 with bad export cookie 4757696470555931137 LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-mdc-ffff8802af83dd28: Connection restored to 192.168.123.128@tcp (at 0@lo) Lustre: Skipped 62 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1992 to 0x0:2017 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2025 to 0x0:2049 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2057 to 0x0:2081 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1993 to 0x0:2017 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9289bf8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 12 previous similar messages Lustre: 28865:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746408756/real 1746408756] req@ffff88032466b240 x1831240112627904/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746408763 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:0.0' Lustre: 28865:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 58 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:2042 to 0x0:2081 Lustre: lustre-OST0003: deleting orphan objects from 0x0:2042 to 0x0:2081 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2074 to 0x0:2113 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2105 to 0x0:2145 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802af83dd28: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.128@tcp) after server handle changed from 0x4206bbbb8ca12fd8 to 0x4206bbbb8ca2bada Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 58 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:2098 to 0x0:2113 Lustre: lustre-OST0003: deleting orphan objects from 0x0:2098 to 0x0:2113 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2163 to 0x0:2209 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2130 to 0x0:2145 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 10 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 12291:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12291:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12291, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802cb7a1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1287 to 0x0:1313 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1286 to 0x0:1313 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1286 to 0x0:1313 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1286 to 0x0:1313 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880296feefc8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1435 to 0x0:1473 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1434 to 0x0:1473 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1434 to 0x0:1473 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1434 to 0x0:1473 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802cb7a1bf8: operation mds_sync to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 21709:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802e8841f80 x1831239527564416/t94489280711(94489280711) o101->lustre-MDT0000-mdc-ffff880296feefc8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1746407956 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 21709:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 363 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1564 to 0x0:1601 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1564 to 0x0:1601 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1564 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1564 to 0x0:1601 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802cb7a1bf8: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1709 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1709 to 0x0:1729 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1708 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1709 to 0x0:1729 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880296feefc8: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: deleting orphan objects from 0x0:1839 to 0x0:1857 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1839 to 0x0:1857 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1839 to 0x0:1857 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1838 to 0x0:1857 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880296feefc8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 8072:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 8072:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 8072, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88007b5fae98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 5 previous similar messages LustreError: 30433:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88027bae0680 x1817649072488128/t94489280685(94489280685) o101->lustre-MDT0000-mdc-ffff88029a0b4138@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733447272 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 30433:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1200 to 0x0:1217 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1296 to 0x0:1313 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1264 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1199 to 0x0:1281 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88007b5fae98: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: MGC192.168.123.188@tcp: Connection restored to 192.168.123.188@tcp (at 0@lo) Lustre: Skipped 36 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1342 to 0x0:1377 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1382 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1349 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1295 to 0x0:1313 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88029a0b4138: operation mds_hsm_state_set to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1396 to 0x0:1441 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1448 to 0x0:1473 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1449 to 0x0:1473 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1489 to 0x0:1505 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1569 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1537 to 0x0:1569 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1506 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1537 to 0x0:1569 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-mdc-ffff88007b5fae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 42 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 7 previous similar messages LustreError: 166-1: MGC192.168.123.188@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: Evicted from MGS (at 192.168.123.188@tcp) after server handle changed from 0x575bff50e8afc2b1 to 0x575bff50e8b117c7 Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1641 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1705 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1662 to 0x0:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1665 to 0x0:1697 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88029a0b4138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: 30439:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1733447532/real 1733447532] req@ffff8802e53bac00 x1817649075289472/t0(0) o400->MGC192.168.123.188@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1733447539 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:3.0' Lustre: 30439:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 123 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1752 to 0x0:1793 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1769 to 0x0:1793 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1819 to 0x0:1857 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1785 to 0x0:1825 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 8928:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 8928:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LustreError: 9655:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x864:0x0].0x0 (ffff8802a0529440) refcount nonzero (2) after lock cleanup; forcing cleanup. Lustre: lustre-OST0003: deleting orphan objects from 0x0:1931 to 0x0:1953 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1866 to 0x0:1889 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1899 to 0x0:1921 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1866 to 0x0:1889 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: lustre-OST0000: deleting orphan objects from 0x0:1967 to 0x0:1985 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1995 to 0x0:2017 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1927 to 0x0:1953 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1930 to 0x0:1953 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times | Link to test |
| replay-dual test 19: resend of open request | LustreError: 21714:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 21714:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 21714, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 16554:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802e6eb25c0 x1817638283264384/t77309411441(0) o101->e22728f7-c71e-4fa6-a043-d93c96e5ccb1@0@lo:223/0 lens 664/656 e 0 to 0 dl 1733436433 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client e22728f7-c71e-4fa6-a043-d93c96e5ccb1 (at 0@lo) reconnecting Lustre: 13337:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802a7464b40 x1817638283264384/t77309411441(0) o101->e22728f7-c71e-4fa6-a043-d93c96e5ccb1@0@lo:230/0 lens 664/3424 e 0 to 0 dl 1733436440 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' | Link to test |
| replay-dual test 19: resend of open request | LustreError: 449:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 449:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 449, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 8771:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802ab167700 x1817569571369280/t77309411443(0) o101->898e1dc0-4211-4561-b9d8-0dddf09d5dc7@0@lo:365/0 lens 664/656 e 0 to 0 dl 1733370890 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client 898e1dc0-4211-4561-b9d8-0dddf09d5dc7 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802e1ce6fc8: Connection restored to (at 0@lo) Lustre: Skipped 18 previous similar messages Lustre: 8775:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008f127700 x1817569571369280/t77309411443(0) o101->898e1dc0-4211-4561-b9d8-0dddf09d5dc7@0@lo:372/0 lens 664/3424 e 0 to 0 dl 1733370897 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' | Link to test |
| replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply | LustreError: 20827:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 20827:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 20827, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 16109:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802e18625c0 x1817546084450496/t4294967324(0) o36->ee0e71ef-b675-4423-81f4-e3691775a258@0@lo:348/0 lens 560/448 e 0 to 0 dl 1733348978 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 16109:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009ed8d7c0 x1817546084450496/t4294967324(0) o36->ee0e71ef-b675-4423-81f4-e3691775a258@0@lo:368/0 lens 560/448 e 0 to 0 dl 1733348998 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:3 to 0x340000401:33 Lustre: DEBUG MARKER: centos-196.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| recovery-small test 27: fail LOV while using OSC's | LustreError: 15571:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 15571:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 15571, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Failing over lustre-MDT0000 LustreError: 30380:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000406:0x18:0x0].0xc42846bf (ffff8802c7df96c0) refcount nonzero (1) after lock cleanup; forcing cleanup. | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 17308:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 17308:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 17308, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802e3f9d3d8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: 30428:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802c49112a8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d6536580/0x8c73a6849bae8c00 lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x8c73a6849bae8bf2 expref: 2 pid: 30428 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x0:1279 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1279 to 0x0:1313 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1278 to 0x0:1313 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1279 to 0x0:1313 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88029f44ae98: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1404 to 0x0:1441 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1405 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1404 to 0x0:1441 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1404 to 0x0:1441 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 8093:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802e57eea80 x1817532792752640/t94489280950(94489280950) o101->lustre-MDT0000-mdc-ffff8802e3f9d3d8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733336162 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 8093:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 351 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1556 to 0x0:1601 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1557 to 0x0:1601 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1557 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1558 to 0x0:1601 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 32291:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802998a4a88 ns: mdt-lustre-MDT0000_UUID lock: ffff88032703b880/0x8c73a6849bb3177a lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x13c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x8c73a6849bb3176c expref: 2 pid: 32291 timeout: 0 lvb_type: 0 LustreError: 32702:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x568:0x0].0x0 (ffff88008e248f40) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 31797:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1733336173 with bad export cookie 10120615876109643948 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff8802e3f9d3d8: operation mds_reint to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 2509:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 2509:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 2509, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 23 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.148@tcp) after server handle changed from 0x9c5cb0deae09628 to 0x9c5cb0deae11f73 Lustre: Skipped 13 previous similar messages LustreError: 6960:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802d4916a80 x1817527628135616/t154618822762(154618822762) o101->lustre-MDT0000-mdc-ffff880291bc9bf8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733331911 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 6960:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 10 previous similar messages Lustre: 7373:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 582, rollback = 2 Lustre: 7373:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 Lustre: 7373:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 7373:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 7373:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 7373:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1642 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1707 to 0x0:1729 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:229 to 0x380000401:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:230 to 0x340000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:229 to 0x300000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:293 to 0x2c0000401:321 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 7040:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880098e18958: can't stat MDS #0: rc = -107 LustreError: 7040:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880098e18958: can't stat MDS #0: rc = -19 LustreError: 7040:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 6 previous similar messages LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:356 to 0x2c0000400:385 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:292 to 0x340000400:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:292 to 0x380000400:321 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:291 to 0x300000400:321 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1705 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1705 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1705 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1768 to 0x0:1793 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times | Link to test |
| replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 | LustreError: 28252:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 28252:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 28252, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 12: open resend timeout | LustreError: 12001:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12001:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12001, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 2 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 5848:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5848:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5848, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d3b1d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1267 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1266 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1042 to 0x0:1057 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1330 to 0x0:1345 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 10980:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007ad63880 x1817501332325952/t94489280689(94489280689) o101->lustre-MDT0000-mdc-ffff880274b52548@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733306361 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 10980:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 112 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1400 to 0x0:1441 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1112 to 0x0:1153 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1369 to 0x0:1409 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1338 to 0x0:1377 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1481 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1224 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1512 to 0x0:1537 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1448 to 0x0:1473 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation mds_reint to node 0@lo failed: rc = -19 Lustre: MGC192.168.123.173@tcp: Connection restored to 192.168.123.173@tcp (at 0@lo) Lustre: Skipped 49 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1335 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1591 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1560 to 0x0:1601 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1624 to 0x0:1665 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ffff880274b52548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 42 previous similar messages Lustre: 10986:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1733306499/real 1733306499] req@ffff88008f925180 x1817501334552064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1733306506 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:3.0' Lustre: 10986:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 65 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1454 to 0x0:1473 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1678 to 0x0:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1743 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1743 to 0x0:1761 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1838 to 0x0:1857 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1550 to 0x0:1569 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1774 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1838 to 0x0:1857 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 19750:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 19750:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 8 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation mds_hsm_state_set to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.123.173@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: Evicted from MGS (at 192.168.123.173@tcp) after server handle changed from 0x9d602335aefe3360 to 0x9d602335aeff202c Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1855 to 0x0:1889 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1632 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1920 to 0x0:1953 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1920 to 0x0:1953 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1962 to 0x0:1985 Lustre: lustre-OST0001: deleting orphan objects from 0x0:2025 to 0x0:2049 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1737 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2026 to 0x0:2049 Lustre: DEBUG MARKER: centos-171.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880274b52548: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages | Link to test |
| replay-dual test 32: gap in update llog shouldn't break recovery | LustreError: 32463:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 32463:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 32463, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=131d, val=0*** Lustre: *** cfs_fail_loc=131d, val=0*** Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 17 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 3693:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 3693:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 3693, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1030 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1669 to 0x0:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1573 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1637 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1605 to 0x0:1633 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 22435:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 22435:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 22435, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11568:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status -116, old was 0 req@ffff8802430a1300 x1817428526325312/t154618822716(154618822716) o35->lustre-MDT0000-mdc-ffff88024dc2efc8@0@lo:23/10 lens 392/456 e 1 to 0 dl 1733237679 ref 2 fl Interpret:RQU/4/0 rc -116/-116 job:'cp.0' LustreError: 11568:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1635 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1633 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1699 to 0x0:1729 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1148 previous similar messages Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.183@tcp (at 0@lo) Lustre: Skipped 142 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:202 to 0x340000401:225 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:203 to 0x380000401:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:203 to 0x300000401:225 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:267 to 0x2c0000401:289 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:28 to 0x300000400:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:25 to 0x380000400:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:29 to 0x340000400:65 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-OST0003: deleting orphan objects from 0x0:1571 to 0x0:1601 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1761 to 0x0:1793 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1696 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1664 to 0x0:1697 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 124 previous similar messages Lustre: 17134:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802ac8ef0c0 x1817428527524096/t34359739150(0) o36->45fcd56b-6ade-4c8d-b42c-17dadf17787d@0@lo:301/0 lens 488/3152 e 0 to 0 dl 1733237946 ref 1 fl Interpret:/2/0 rc 0/0 job:'tar.0' Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:254 to 0x300000401:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:318 to 0x2c0000401:353 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:254 to 0x340000401:289 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:255 to 0x380000401:289 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:149 to 0x380000400:193 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:149 to 0x340000400:193 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:92 to 0x2c0000400:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:150 to 0x300000400:193 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LustreError: 11-0: MGC192.168.123.183@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 LustreError: Skipped 31 previous similar messages LustreError: 166-1: MGC192.168.123.183@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1806 to 0x0:1825 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1613 to 0x0:1633 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1741 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1710 to 0x0:1729 Lustre: DEBUG MARKER: centos-181.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 3125:0:(osd_handler.c:698:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** LustreError: 3125:0:(osd_handler.c:698:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: Skipped 12 previous similar messages | Link to test |
| replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update | LustreError: 27762:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27762:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27762, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 10346:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802eba79940 x1817419467017216/t107374182411(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:581/0 lens 2200/4320 e 0 to 0 dl 1733228411 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' LustreError: 4731:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880095d5cb40 x1817419467016192/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff8802c812d3d8@0@lo:12/10 lens 648/600 e 0 to 0 dl 1733228506 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'lfs.0' LustreError: 4731:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 2 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1187 to 0x0:1377 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1161 to 0x0:1409 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:904 to 0x0:1185 Lustre: DEBUG MARKER: centos-231.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| sanity-pfl test 20d: Low on space + 0-length comp: force extension | LustreError: 7744:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 7744:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 7744, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| sanity test 160h: changelog gc thread stop upon umount, orphan records delete | LustreError: 9754:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 9754:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 9754, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1316, val=0*** Lustre: 15967:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: simulate starting changelog garbage collection Lustre: 15967:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 2 previous similar messages Lustre: 8651:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl11 idle for 15s with 3 unprocessed records Lustre: 8651:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 2 previous similar messages Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-lwp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -19 Lustre: 8707:0:(lwp_dev.c:182:lwp_disconnect()) lustre-MDT0000-lwp-MDT0002: can't disconnect: rc = -19 LustreError: 8707:0:(obd_mount_server.c:989:lustre_disconnect_lwp()) lustre-MDT0000-lwp-MDT0002: fail to disconnect LWP: rc = -19 Lustre: 8707:0:(obd_mount_server.c:1603:server_put_super()) lustre-MDT0002: failed to disconnect lwp: rc= -19 LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: Failing over lustre-MDT0001 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 9: resending a replayed create | LustreError: 10926:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 10926:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 10926, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 2298:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 2298:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 11: both clients timeout during replay | LustreError: 15254:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 15254:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 15254, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update | LustreError: 12346:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12346:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12346, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 27885:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff880090ab4b40 x1817363501161408/t107374182411(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:455/0 lens 2200/4320 e 0 to 0 dl 1733174680 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' LustreError: 22105:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008a01a5c0 x1817363501160384/t107374182410(107374182410) o101->lustre-MDT0000-mdc-ffff8802c8e84a88@0@lo:12/10 lens 648/600 e 0 to 0 dl 1733174734 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'lfs.0' LustreError: 22105:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1377 Lustre: lustre-OST0003: deleting orphan objects from 0x0:937 to 0x0:1185 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1187 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1065 to 0x0:1313 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 2047:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 2047:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 2047, comm: kworker/u32:6 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8803246192a8: operation mds_close to node 0@lo failed: rc = -107 LustreError: 16193:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b5adbec0 x1817342275117056/t0(0) o2->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-2-0.0' Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1249 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1249 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1313 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1248 to 0x0:1281 Lustre: DEBUG MARKER: centos-206.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 7 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff88008f792e98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 7 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1481 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1417 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1417 to 0x0:1441 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1416 to 0x0:1441 Lustre: DEBUG MARKER: centos-206.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88008f792e98: operation mds_hsm_state_set to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 16204:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ea239300 x1817342276968384/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' LustreError: 16204:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 16189:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802a4c1d180 x1817342274811968/t94489280759(94489280759) o101->lustre-MDT0000-mdc-ffff8803246192a8@0@lo:12/10 lens 648/600 e 0 to 0 dl 1733154339 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 16189:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 404 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1634 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1569 to 0x0:1601 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1569 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1569 to 0x0:1601 Lustre: DEBUG MARKER: centos-206.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 19: resend of open request | LustreError: 2266:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 2266:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 2266, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 23526:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8800b2c53240 x1817330271456512/t77309411441(0) o101->c05d0527-90b7-432b-8b32-a514bc49baff@0@lo:213/0 lens 664/656 e 0 to 0 dl 1733142728 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client c05d0527-90b7-432b-8b32-a514bc49baff (at 0@lo) reconnecting Lustre: 3863:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802a0dec500 x1817330271456512/t77309411441(0) o101->c05d0527-90b7-432b-8b32-a514bc49baff@0@lo:220/0 lens 664/3424 e 0 to 0 dl 1733142735 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages | Link to test |
| replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 | LustreError: 9407:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 9407:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 9407, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 2575:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802ea578680 x1817314070801792/t137438953491(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:751/0 lens 1488/4320 e 0 to 0 dl 1733128166 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x0:1411 to 0x0:1537 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1475 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1065 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1569 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| recovery-small test 28: handle error adding new clients (bug 6086) | LustreError: 836:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 836:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 836, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=305, val=0*** Lustre: Skipped 25 previous similar messages Lustre: 20426:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1733114188/real 1733114188] req@ffff8802d8e83ec0 x1817300566168960/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 328/224 e 0 to 1 dl 1733114199 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' Lustre: 20426:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Lustre: *** cfs_fail_loc=12f, val=0*** LustreError: 29449:0:(tgt_lastrcvd.c:1028:tgt_client_new()) lustre-OST0001: no room for 4 clients - fix LR_MAX_CLIENTS LustreError: 11-0: lustre-OST0001-osc-ffff8800a53a8958: operation ost_connect to node 0@lo failed: rc = -75 LustreError: Skipped 3 previous similar messages Lustre: Failing over lustre-MDT0000 | Link to test |
| recovery-small test 106: lightweight connection support | LustreError: 24620:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 24620:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 24620, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=805, val=0*** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 5025:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5025:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5025, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 30830:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802c4f3b240 x1817220216137984/t154618822737(154618822737) o101->lustre-MDT0000-mdc-ffff8802dfb00958@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733038939 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 30830:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: 31399:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 582, rollback = 2 Lustre: 31399:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/0, destroy: 0/0/0 Lustre: 31399:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 31399:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 31399:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 31399:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 1602:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802ceb74b40 x1817220216321856/t154618822932(0) o36->3fc95299-1bc8-4676-a8b4-ac54c588ca79@0@lo:583/0 lens 504/448 e 0 to 0 dl 1733038908 ref 1 fl Interpret:/2/0 rc 0/0 job:'tar.0' Lustre: lustre-OST0003: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1644 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1707 to 0x0:1729 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 9 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1109 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: 32029:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 582, rollback = 2 Lustre: 32029:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 23 previous similar messages Lustre: 32029:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/2, destroy: 0/0/0 Lustre: 32029:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 32029:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 32029:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 32029:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 32029:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 32029:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/4, delete: 0/0/0 Lustre: 32029:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 32029:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 32029:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:235 to 0x300000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:299 to 0x2c0000401:321 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:235 to 0x340000401:257 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:235 to 0x380000401:257 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 30840:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b2145e00 x1817220218561472/t0(0) o2->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-2.0' LustreError: 1585:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88029162ae98 ns: mdt-lustre-MDT0002_UUID lock: ffff8802e53cc780/0xffaa0f357b249755 lrc: 3/0,0 mode: PR/PR res: [0x280000405:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xffaa0f357b249747 expref: 9 pid: 1585 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:263 to 0x380000400:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:327 to 0x2c0000400:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:262 to 0x300000400:289 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:262 to 0x340000400:289 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x0:1692 to 0x0:1729 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1757 to 0x0:1793 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1693 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1693 to 0x0:1729 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times LustreError: 1306:0:(ldlm_resource.c:1126:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200012512:0xc:0x0].0x0 (ffff8802fcfa1bc0) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 1306:0:(ldlm_resource.c:1126:ldlm_resource_complain()) Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.193@tcp (at 0@lo) Lustre: Skipped 151 previous similar messages Lustre: 1499:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 511 < left 582, rollback = 2 Lustre: 1499:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 17 previous similar messages Lustre: 1499:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/2, destroy: 0/0/0 Lustre: 1499:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 1499:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 1499:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 1499:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 1499:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 1499:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/2, delete: 0/0/0 Lustre: 1499:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 1499:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 1499:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 1600:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c4e48040 x1817220220062080/t34359740356(0) o36->3fc95299-1bc8-4676-a8b4-ac54c588ca79@0@lo:78/0 lens 488/3152 e 0 to 0 dl 1733039158 ref 1 fl Interpret:/2/0 rc 0/0 job:'tar.0' Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:305 to 0x300000401:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:304 to 0x380000401:321 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:369 to 0x2c0000401:385 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:304 to 0x340000401:321 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 11-0: lustre-MDT0002-mdc-ffff8800ab044a88: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 25 previous similar messages Lustre: lustre-MDT0002-mdc-ffff8800ab044a88: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 131 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 7923:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 7923:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 7923, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a5c18958: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: 10383:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8800a5c1ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ec6634c0/0x812af00b3b7ad2ad lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x812af00b3b7ad29f expref: 3 pid: 10383 timeout: 0 lvb_type: 0 Lustre: MGC192.168.123.158@tcp: Connection restored to 192.168.123.158@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1198 to 0x0:1217 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1198 to 0x0:1217 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1294 to 0x0:1313 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1262 to 0x0:1281 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880259beca88: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: deleting orphan objects from 0x0:1355 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1286 to 0x0:1313 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1285 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1388 to 0x0:1409 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a5c18958: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-OST0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 39 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1462 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1399 to 0x0:1441 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1494 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1399 to 0x0:1441 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: 24686:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1733024729/real 1733024729] req@ffff8802967057c0 x1817205786186880/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1733024773 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:0.0' Lustre: 24686:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880259beca88: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 24794:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x480:0x0].0x0 (ffff88009054e840) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-OST0003: deleting orphan objects from 0x0:1619 to 0x0:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1531 to 0x0:1601 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1529 to 0x0:1569 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1601 to 0x0:1633 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880259beca88: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 24672:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802845bb240 x1817205786419904/t107374183104(107374183104) o101->lustre-MDT0000-mdc-ffff880259beca88@0@lo:12/10 lens 576/600 e 0 to 0 dl 1733024922 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 24672:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 428 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1712 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1681 to 0x0:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1766 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1641 to 0x0:1665 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880259beca88: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1720 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1792 to 0x0:1825 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1753 to 0x0:1793 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1853 to 0x0:1889 Lustre: DEBUG MARKER: centos-156.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 8 previous similar messages | Link to test |
| replay-dual test 8: replay of resent request | LustreError: 12399:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12399:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12399, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 5697:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802958f4b40 x1817193878886272/t38654705670(0) o36->4a2dd7a6-cebb-4f9f-b9c8-a3fcac3b8338@0@lo:632/0 lens 512/448 e 0 to 0 dl 1733011777 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' Lustre: lustre-MDT0000-mdc-ffff8800988ab7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Client 4a2dd7a6-cebb-4f9f-b9c8-a3fcac3b8338 (at 0@lo) reconnecting Lustre: 5698:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802f66a0cc0 x1817193878886272/t38654705670(0) o36->4a2dd7a6-cebb-4f9f-b9c8-a3fcac3b8338@0@lo:639/0 lens 512/448 e 0 to 0 dl 1733011784 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' | Link to test |
| replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply | LustreError: 27196:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27196:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27196, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 18884:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802b5cb5180 x1817177346325312/t8589934616(0) o36->e17ec183-8a96-4c49-b30c-341d4f7987ff@0@lo:380/0 lens 560/448 e 0 to 0 dl 1732997180 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages | Link to test |
| recovery-small test 54: back in time | LustreError: 5707:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5707:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5707, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 5993:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5993:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5993, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802ccd7dd28: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages LustreError: 21407:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880279bff0c0 x1817145030383040/t94489280753(94489280753) o101->lustre-MDT0000-mdc-ffff88029db253d8@0@lo:12/10 lens 648/600 e 0 to 0 dl 1732966404 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 21407:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 2 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1203 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1299 to 0x0:1345 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1235 to 0x0:1281 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1235 to 0x0:1281 Lustre: DEBUG MARKER: centos-231.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88029db253d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 | Link to test |
| recovery-small test 23: client hang when close a file after mds crash | LustreError: 27984:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27984:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27984, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 12947:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12947:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12947, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 4590:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802a463ac00 x1817119754430912/t94489280754(94489280754) o101->lustre-MDT0000-mdc-ffff8802a3502548@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732942195 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 4590:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: lustre-OST0000: deleting orphan objects from 0x0:1290 to 0x0:1313 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1226 to 0x0:1249 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1227 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1226 to 0x0:1249 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802a2ca1bf8: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 9: resending a replayed create | LustreError: 6119:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 6119:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 6119, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 10415:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 10415:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 30203:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 30203:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 30203, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-OST0003: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1700 to 0x0:1729 Lustre: DEBUG MARKER: centos-221.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.223@tcp (at 0@lo) Lustre: Skipped 164 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:210 to 0x380000401:225 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:217 to 0x340000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:218 to 0x300000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:281 to 0x2c0000401:321 Lustre: DEBUG MARKER: centos-221.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 4587:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880248f8efc8: can't stat MDS #0: rc = -19 LustreError: 4587:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880248f8efc8: can't stat MDS #0: rc = -19 LustreError: 4587:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 3 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:21 to 0x380000400:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:22 to 0x300000400:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:85 to 0x2c0000400:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:22 to 0x340000400:65 Lustre: DEBUG MARKER: centos-221.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 26374:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 26374:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 26374, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 23238:0:(ldlm_resource.c:1126:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x2400032e1:0x11:0x0].0x0 (ffff88008d44cf40) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages BUG: sleeping function called from invalid context at kernel/rwsem.c:51 in_atomic(): 1, irqs_disabled(): 0, pid: 23238, name: umount CPU: 8 PID: 23238 Comm: umount Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffff810c8ec9>] __might_sleep+0xd9/0x100 [<ffffffff817dfe40>] down_write+0x20/0x50 [<ffffffffa1045227>] osp_invalidate+0x177/0x220 [osp] [<ffffffffa105cf83>] osp_trans_stop_cb+0x133/0x180 [osp] [<ffffffffa1060047>] osp_trans_callback+0xa7/0xc0 [osp] [<ffffffffa1039ef0>] osp_update_fini.isra.11+0xa0/0x270 [osp] [<ffffffffa103a356>] osp_process_config+0x296/0x590 [osp] [<ffffffffa0f48b3d>] lod_sub_process_config+0x8d/0x1d0 [lod] [<ffffffffa0f500e8>] lod_process_config+0x558/0x1390 [lod] [<ffffffffa0ddf498>] mdd_process_config+0x88/0x610 [mdd] [<ffffffffa0e598cb>] mdt_stack_pre_fini+0x26b/0x6f0 [mdt] [<ffffffffa0e5ae21>] mdt_device_fini+0x81/0x990 [mdt] [<ffffffffa03cad4c>] class_cleanup+0xa3c/0xd00 [obdclass] [<ffffffffa03cbbd7>] class_process_config+0x527/0x26e0 [obdclass] [<ffffffffa03cdf56>] class_manual_cleanup+0x1c6/0x760 [obdclass] [<ffffffffa0408975>] server_put_super+0xa25/0xf80 [obdclass] [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff81291746>] ? fsnotify_unmount_inodes+0x1d6/0x1e0 [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1698a9b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17 BUG: scheduling while atomic: umount/23238/0x10000002 Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw i2c_core libata [last unloaded: libcfs] CPU: 8 PID: 23238 Comm: umount Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffff817d2b8e>] __schedule_bug+0x64/0x72 [<ffffffff817e0a31>] __schedule+0x791/0x7d0 [<ffffffff810cc9c6>] __cond_resched+0x26/0x30 [<ffffffff817e0d4a>] _cond_resched+0x3a/0x50 [<ffffffff817dfe45>] down_write+0x25/0x50 [<ffffffffa1045227>] osp_invalidate+0x177/0x220 [osp] [<ffffffffa105cf83>] osp_trans_stop_cb+0x133/0x180 [osp] [<ffffffffa1060047>] osp_trans_callback+0xa7/0xc0 [osp] [<ffffffffa1039ef0>] osp_update_fini.isra.11+0xa0/0x270 [osp] [<ffffffffa103a356>] osp_process_config+0x296/0x590 [osp] [<ffffffffa0f48b3d>] lod_sub_process_config+0x8d/0x1d0 [lod] [<ffffffffa0f500e8>] lod_process_config+0x558/0x1390 [lod] [<ffffffffa0ddf498>] mdd_process_config+0x88/0x610 [mdd] [<ffffffffa0e598cb>] mdt_stack_pre_fini+0x26b/0x6f0 [mdt] [<ffffffffa0e5ae21>] mdt_device_fini+0x81/0x990 [mdt] [<ffffffffa03cad4c>] class_cleanup+0xa3c/0xd00 [obdclass] [<ffffffffa03cbbd7>] class_process_config+0x527/0x26e0 [obdclass] [<ffffffffa03cdf56>] class_manual_cleanup+0x1c6/0x760 [obdclass] [<ffffffffa0408975>] server_put_super+0xa25/0xf80 [obdclass] [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff81291746>] ? fsnotify_unmount_inodes+0x1d6/0x1e0 [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1698a9b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17 LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages LustreError: 23019:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802a8ada5c0 x1817074801746432/t154618822766(154618822766) o101->lustre-MDT0000-mdc-ffff8802c7b512a8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732900250 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 23019:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: 23457:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 582, rollback = 2 Lustre: 23457:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 Lustre: 23457:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 23457:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 23457:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 23457:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1644 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1644 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1708 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1644 to 0x0:1665 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 21 previous similar messages LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8800941add28: can't stat MDS #0: rc = -107 LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8800941add28: can't stat MDS #0: rc = -19 LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 13 previous similar messages LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8800941add28: can't stat MDS #0: rc = -19 LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 17 previous similar messages LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8800941add28: can't stat MDS #0: rc = -19 LustreError: 23111:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 9 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:238 to 0x380000401:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:237 to 0x340000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:238 to 0x300000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:302 to 0x2c0000400:321 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:35 to 0x340000400:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:99 to 0x2c0000401:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:36 to 0x300000400:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:35 to 0x380000400:65 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1008 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1701 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1700 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1700 to 0x0:1729 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1765 to 0x0:1793 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages LustreError: 32561:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88009dee0008 ns: mdt-lustre-MDT0001_UUID lock: ffff8802bef4f0c0/0x6543d261dc96c28f lrc: 3/0,0 mode: PR/PR res: [0x240003ab0:0x18:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x6543d261dc96c281 expref: 2153 pid: 32561 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.228@tcp (at 0@lo) Lustre: Skipped 169 previous similar messages Lustre: 25970:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 582, rollback = 2 Lustre: 25970:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 35 previous similar messages Lustre: 25970:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/0, destroy: 0/0/0 Lustre: 25970:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25970:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 25970:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25970:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 25970:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25970:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 25970:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25970:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 25970:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:599 to 0x380000401:641 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:662 to 0x2c0000400:705 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:598 to 0x300000401:641 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:598 to 0x340000401:641 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 26175:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732900425 with bad export cookie 7296907139024774154 LustreError: 26175:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 1 previous similar message LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 26698:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 511 < left 582, rollback = 2 Lustre: 26698:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages Lustre: 26698:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/1, destroy: 0/0/0 Lustre: 26698:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 5 previous similar messages Lustre: 26698:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 26698:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 5 previous similar messages Lustre: 26698:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 26698:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 5 previous similar messages Lustre: 26698:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/3, delete: 0/0/0 Lustre: 26698:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 5 previous similar messages Lustre: 26698:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 26698:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 5 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:134 to 0x380000400:161 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:135 to 0x300000400:161 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:134 to 0x340000400:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:198 to 0x2c0000401:225 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.228@tcp) after server handle changed from 0x6543d261dc9563af to 0x6543d261dc9ad76b Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 15 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: Skipped 15 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1836 to 0x0:1857 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1771 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1771 to 0x0:1793 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1771 to 0x0:1793 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: Skipped 15 previous similar messages LustreError: 11-0: lustre-MDT0001-mdc-ffff8802c7b512a8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 28 previous similar messages Lustre: lustre-MDT0001-mdc-ffff8802c7b512a8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 138 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: Skipped 15 previous similar messages LustreError: 23180:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732900535 with bad export cookie 7296907139024954187 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:743 to 0x2c0000400:769 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:683 to 0x380000401:705 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:680 to 0x340000401:705 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:683 to 0x300000401:705 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times | Link to test |
| recovery-small test 29a: error adding new clients doesn't cause LBUG (bug 22273) | LustreError: 3017:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 3017:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 3017, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Failing over lustre-MDT0000 | Link to test |
| sanity test 160a: changelog sanity | LustreError: 29985:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 29985:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 29985, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: lustre-MDD0000: changelog on Lustre: lustre-MDD0002: changelog on Lustre: Skipped 1 previous similar message Lustre: Failing over lustre-MDT0000 | Link to test |
| recovery-small test 54: back in time | LustreError: 22698:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 22698:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 22698, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Mounted lustre-client | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 9094:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 9094:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 9094, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d44aca88: operation mds_close to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1329 to 0x0:1345 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1265 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1266 to 0x0:1281 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1202 to 0x0:1217 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d44aca88: operation mds_sync to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1372 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1373 to 0x0:1441 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1436 to 0x0:1473 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1308 to 0x0:1345 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d44aca88: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 3159:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802c1ee37e8 ns: mdt-lustre-MDT0000_UUID lock: ffff880323f68040/0x3a850c543ea54109 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0xe4:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x3a850c543ea540fb expref: 3 pid: 3159 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 12230:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802ef6057c0 x1817042128357120/t94489280768(94489280768) o101->lustre-MDT0000-mdc-ffff8802d44aca88@0@lo:12/10 lens 648/600 e 0 to 0 dl 1732868245 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 12230:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 314 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1584 to 0x0:1601 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1519 to 0x0:1537 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1552 to 0x0:1569 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1456 to 0x0:1473 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d44aca88: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 3282:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 3282:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 3282, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bc8d92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1263 to 0x0:1281 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1167 to 0x0:1185 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1263 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1187 to 0x0:1217 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a4f1c138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1265 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1370 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1308 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1367 to 0x0:1409 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a4f1c138: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages LustreError: 7643:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8800a6e84a88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802618d1a80/0x1f42e72de847147a lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1f42e72de847146c expref: 3 pid: 7643 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.128@tcp (at 0@lo) Lustre: Skipped 44 previous similar messages LustreError: 14136:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880094521940 x1817001934392832/t94489280783(94489280783) o101->lustre-MDT0000-mdc-ffff8802bc8d92a8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732830251 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 14136:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 196 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1418 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1481 to 0x0:1505 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1478 to 0x0:1505 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1347 to 0x0:1377 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bc8d92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 13536:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x48f:0x0].0x0 (ffff88031d6c4cc0) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: deleting orphan objects from 0x0:1588 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1577 to 0x0:1601 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1461 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1528 to 0x0:1569 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bc8d92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ffff8802bc8d92a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 49 previous similar messages LustreError: 14434:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8800ab5ec138 ns: mdt-lustre-MDT0000_UUID lock: ffff880099524780/0x1f42e72de848fd85 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0xbb:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1f42e72de848fd77 expref: 4 pid: 14434 timeout: 0 lvb_type: 0 LustreError: 14145:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802a2818040 x1817001936531072/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: 14149:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732830309/real 1732830309] req@ffff880083b16440 x1817001936534144/t0(0) o400->MGC192.168.123.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1732830316 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:2.0' Lustre: 14149:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 93 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1701 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1654 to 0x0:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1571 to 0x0:1601 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bc8d92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1818 to 0x0:1857 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1790 to 0x0:1825 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1690 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1762 to 0x0:1793 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 24038:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 24038:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 8 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bc8d92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message | Link to test |
| recovery-small test 106: lightweight connection support | LustreError: 15324:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 15324:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 15324, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=805, val=0*** LustreError: 5043:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| conf-sanity test 84: check recovery_hard_time | LustreError: 7903:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 7903:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 7903, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LDISKFS-fs (dm-4): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-4): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 15090:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 15090:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 15090, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1194 to 0x0:1217 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1226 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1291 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1227 to 0x0:1249 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 25360:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88029d193240 x1816909196209024/t94489280770(94489280770) o101->lustre-MDT0000-mdc-ffff8800ac7637e8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732741602 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 25360:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 73 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1276 to 0x0:1313 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1381 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1321 to 0x0:1345 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1316 to 0x0:1345 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0002: deleting orphan objects from 0x0:1420 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1389 to 0x0:1409 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1471 to 0x0:1505 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1415 to 0x0:1441 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88029c0192a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: MGC192.168.123.163@tcp: Connection restored to 192.168.123.163@tcp (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1467 to 0x0:1505 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1499 to 0x0:1537 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1499 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1562 to 0x0:1601 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1593 to 0x0:1633 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1627 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1617 to 0x0:1633 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1692 to 0x0:1729 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1821 to 0x0:1857 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1724 to 0x0:1761 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1757 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1724 to 0x0:1761 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac7637e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message | Link to test |
| replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir | LustreError: 15017:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 15017:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 15017, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 8865:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 29289:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 29289:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 29289, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802e5918008: operation mds_close to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1285 to 0x0:1313 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1220 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: DEBUG MARKER: centos-221.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88008f0c5d28: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 31376:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8803218de440 x1816887458732224/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' | Link to test |
| replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply | LustreError: 6892:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 6892:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 6892, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 26359:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802c0173240 x1816886487045184/t4294967324(0) o36->fa9cfa2d-6754-4a4f-a30d-99b292a21605@0@lo:37/0 lens 560/448 e 0 to 0 dl 1732719752 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 7 previous similar messages LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 6 previous similar messages Lustre: 26360:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802b94c9300 x1816886487045184/t4294967324(0) o36->fa9cfa2d-6754-4a4f-a30d-99b292a21605@0@lo:57/0 lens 560/448 e 0 to 0 dl 1732719772 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:3 to 0x340000401:33 Lustre: DEBUG MARKER: centos-126.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 2596:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 2596:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 2596, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] LustreError: 6808:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 6808:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe Pid: 6808, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1635 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1668 to 0x0:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1700 to 0x0:1729 Lustre: DEBUG MARKER: centos-166.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.168@tcp (at 0@lo) Lustre: Skipped 164 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:269 to 0x2c0000400:289 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:203 to 0x340000401:225 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:202 to 0x380000401:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:171 to 0x300000401:193 Lustre: DEBUG MARKER: centos-166.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 979 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:27 to 0x300000400:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:92 to 0x2c0000401:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:27 to 0x340000400:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:28 to 0x380000400:65 Lustre: DEBUG MARKER: centos-166.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 149 previous similar messages | Link to test |
| replay-single test 11: create open write rename |X| create-old-name read | LustreError: 24975:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 24975:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 24975, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1435:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 1435:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 1435, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 7812:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008e788040 x1816807626810624/t154618822707(154618822707) o101->lustre-MDT0000-mdc-ffff8802bfd94138@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732645677 ref 2 fl Interpret:RQU/4/0 rc 301/301 job:'tar.0' LustreError: 7812:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1731 to 0x0:1761 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1667 to 0x0:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1667 to 0x0:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1633 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.163@tcp (at 0@lo) Lustre: Skipped 153 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:202 to 0x380000401:225 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:202 to 0x340000401:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:203 to 0x300000401:225 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:267 to 0x2c0000401:289 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 1 previous similar message LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 3 previous similar messages LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 6 previous similar messages LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 857:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 28 previous similar messages Lustre: lustre-MDT0002: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:90 to 0x2c0000400:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:27 to 0x340000400:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:28 to 0x380000400:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:25 to 0x300000400:65 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880093ad2548: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 29 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 15 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: Skipped 15 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1721 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1786 to 0x0:1825 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1723 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1659 to 0x0:1697 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 17772:0:(osd_handler.c:698:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** LustreError: 17772:0:(osd_handler.c:698:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 143 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: Skipped 15 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:251 to 0x340000401:289 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:251 to 0x300000401:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:318 to 0x2c0000401:353 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:254 to 0x380000401:289 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 13437:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0002-mdc-ffff880093ad2548: can't stat MDS #0: rc = -19 LustreError: 13437:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 18 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:94 to 0x300000400:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:91 to 0x380000400:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:88 to 0x340000400:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:154 to 0x2c0000400:193 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Evicted from MGS (at 192.168.123.163@tcp) after server handle changed from 0x65f0b1b78abd2c39 to 0x65f0b1b78ac1d056 Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1773 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1715 to 0x0:1761 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1777 to 0x0:1793 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1841 to 0x0:1857 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: Skipped 13 previous similar messages LustreError: 32109:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0001_UUID: namespace resource [0x240000403:0x55a:0x0].0x0 (ffff880323ef74c0) refcount nonzero (1) after lock cleanup; forcing cleanup. LustreError: 13274:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732645919 with bad export cookie 7345566394106221653 LustreError: 13274:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:670 to 0x2c0000401:705 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:645 to 0x380000401:673 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:593 to 0x340000401:609 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:634 to 0x300000401:673 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 17353:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 17353:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 17353, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d24befc8: operation ldlm_cancel to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 22195:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88029fc337e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e0efb4c0/0x877235196788a8e5 lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x50:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x877235196788a8d7 expref: 3 pid: 22195 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: deleting orphan objects from 0x0:1254 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1317 to 0x0:1345 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1254 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1254 to 0x0:1281 Lustre: DEBUG MARKER: centos-186.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d24befc8: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 18742:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 18742:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 18742, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88026fa3e678: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 8131:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802670370c0 x1816758883892288/t94489280740(94489280740) o101->lustre-MDT0000-mdc-ffff8802639de678@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732598246 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 8131:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 2 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1215 to 0x0:1249 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1215 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1215 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1279 to 0x0:1313 Lustre: DEBUG MARKER: centos-141.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802639de678: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1409 to 0x0:1441 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1345 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1328 to 0x0:1345 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1340 to 0x0:1409 Lustre: DEBUG MARKER: centos-141.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802639de678: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.143@tcp (at 0@lo) Lustre: Skipped 44 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1505 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1441 to 0x0:1473 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1538 to 0x0:1569 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1474 to 0x0:1505 Lustre: DEBUG MARKER: centos-141.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1581 to 0x0:1601 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1612 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1550 to 0x0:1569 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1645 to 0x0:1665 Lustre: DEBUG MARKER: centos-141.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802639de678: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages | Link to test |
| replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update | LustreError: 4338:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 4338:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 4338, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 24767:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff880087853ec0 x1816739772870272/t115964117001(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:321/0 lens 2200/4320 e 0 to 0 dl 1732580361 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 24617:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732580323 with bad export cookie 1708404097388109376 LustreError: 24617:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 21094:0:(ldlm_resource.c:1126:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000013a1:0x76:0x0].0x0 (ffff880297755440) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-OST0003: deleting orphan objects from 0x0:1065 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1345 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1155 to 0x0:1377 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:100 to 0x2c0000401:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:68 to 0x340000401:97 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:68 to 0x300000401:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:68 to 0x380000401:97 Lustre: 24727:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800adb20040 x1816739772869440/t12884901936(0) o36->bb5eecf2-e5a2-4f83-bc76-20de0d2a0523@0@lo:370/0 lens 560/448 e 0 to 0 dl 1732580410 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply | LustreError: 25563:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 25563:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 25563, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 32084:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802817fea80 x1816701250937728/t8589934616(0) o36->5daadc87-cb8f-4440-a3dd-d978fd5ddb1f@0@lo:171/0 lens 560/448 e 0 to 0 dl 1732543216 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' LustreError: 32046:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732543213 with bad export cookie 15546167582998355543 LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:67 to 0x2c0000401:97 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:35 to 0x300000401:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:35 to 0x380000401:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:35 to 0x340000401:65 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1123 to 0x0:1249 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1065 to 0x0:1281 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1065 to 0x0:1249 Lustre: 32076:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d7ec1940 x1816701250937728/t8589934616(0) o36->5daadc87-cb8f-4440-a3dd-d978fd5ddb1f@0@lo:209/0 lens 560/448 e 0 to 0 dl 1732543254 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' Lustre: DEBUG MARKER: centos-166.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 10414:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 10414:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 10414, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d315ae98: operation ldlm_enqueue to node 0@lo failed: rc = -19 | Link to test |
| recovery-small test 23: client hang when close a file after mds crash | LustreError: 9772:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 9772:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 9772, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 23653:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23653:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23653, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages LustreError: 166-1: MGC192.168.123.238@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 14 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.238@tcp) after server handle changed from 0x61166fd61c0ed23d to 0x61166fd61c0fe0f8 Lustre: Skipped 14 previous similar messages Lustre: 23355:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 505 < left 582, rollback = 2 Lustre: 23355:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 Lustre: 23355:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 23355:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 23355:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/4, delete: 0/0/0 Lustre: 23355:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-MDT0000: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 21 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1650 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1714 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1603 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1650 to 0x0:1665 Lustre: DEBUG MARKER: centos-236.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:230 to 0x300000401:257 Lustre: lustre-OST0003: deleting orphan objects from 0x340000401:230 to 0x340000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:294 to 0x2c0000401:321 Lustre: lustre-OST0002: deleting orphan objects from 0x380000401:230 to 0x380000401:257 Lustre: DEBUG MARKER: centos-236.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 11-0: lustre-MDT0002-mdc-ffff8802ad40e678: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 27 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 23486:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802bb9e4138 ns: mdt-lustre-MDT0002_UUID lock: ffff8802bc157840/0x61166fd61c149f16 lrc: 3/0,0 mode: CW/CW res: [0x280000404:0x14:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x61166fd61c149f08 expref: 4 pid: 23486 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x380000400:333 to 0x380000400:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:332 to 0x300000400:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:396 to 0x2c0000400:417 Lustre: lustre-OST0003: deleting orphan objects from 0x340000400:333 to 0x340000400:353 Lustre: DEBUG MARKER: centos-236.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 25206:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 582, rollback = 2 Lustre: 25206:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 35 previous similar messages Lustre: 25206:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/1, destroy: 0/0/0 Lustre: 25206:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25206:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 25206:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25206:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 25206:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25206:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/2, delete: 0/0/0 Lustre: 25206:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: 25206:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 25206:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 35 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1698 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1666 to 0x0:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1698 to 0x0:1729 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1762 to 0x0:1793 Lustre: DEBUG MARKER: centos-236.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 23021:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802c9091f80 x1816572684641408/t34359739803(34359739803) o101->lustre-MDT0001-mdc-ffff8800a5db6678@0@lo:12/10 lens 648/600 e 0 to 0 dl 1732421202 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 23021:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 180 previous similar messages Lustre: 25870:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 582, rollback = 2 Lustre: 25870:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 17 previous similar messages Lustre: 25870:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/3, destroy: 0/0/0 Lustre: 25870:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 25870:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 25870:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 25870:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 25870:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 25870:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/4, delete: 0/0/0 Lustre: 25870:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: 25870:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 25870:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 17 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x380000401:324 to 0x380000401:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:324 to 0x300000401:353 Lustre: lustre-OST0003: deleting orphan objects from 0x340000401:323 to 0x340000401:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:388 to 0x2c0000401:417 Lustre: DEBUG MARKER: centos-236.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 28741:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802a1b40008 ns: mdt-lustre-MDT0002_UUID lock: ffff8800a4c31a80/0x61166fd61c192526 lrc: 3/0,0 mode: PR/PR res: [0x280000404:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x61166fd61c192518 expref: 52 pid: 28741 timeout: 0 lvb_type: 0 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 29692:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 29692:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 29692, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009ec02e98: operation mds_hsm_state_set to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 5 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1222 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1285 to 0x0:1313 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009ec02e98: operation mds_hsm_state_set to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 16000:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88007a619300 x1816562740426560/t94489280717(94489280717) o101->lustre-MDT0000-mdc-ffff88009ec02e98@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732411467 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 16000:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 70 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1327 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1325 to 0x0:1345 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1390 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1326 to 0x0:1345 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802cac8ae98: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: MGC192.168.123.193@tcp: Connection restored to (at 0@lo) Lustre: Skipped 43 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1416 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1415 to 0x0:1441 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1415 to 0x0:1441 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1479 to 0x0:1505 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009ec02e98: operation mds_hsm_state_set to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 4827:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 4827:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 4827, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d0cc8008: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1221 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1285 to 0x0:1313 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800957dc138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x0:1336 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1335 to 0x0:1377 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1336 to 0x0:1377 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1399 to 0x0:1441 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802d0cc8008: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 22066:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f44d9f80 x1816560860814656/t0(0) o2->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-1-0.0' LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 22063:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a9d51940 x1816560859225408/t94489280719(94489280719) o101->lustre-MDT0000-mdc-ffff8802d0cc8008@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732409384 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 22063:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 284 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1499 to 0x0:1537 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1499 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1499 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1562 to 0x0:1601 Lustre: DEBUG MARKER: centos-146.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 11946:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 11946:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 11946, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880262f6d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1257 to 0x0:1281 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1193 to 0x0:1217 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1289 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1257 to 0x0:1281 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880262f6d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 22499:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802d3f15e00 x1816554541596352/t94489280684(94489280684) o101->lustre-MDT0000-mdc-ffff880262f6d3d8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732403575 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 22499:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 96 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1344 to 0x0:1377 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1326 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1277 to 0x0:1313 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1374 to 0x0:1409 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880323da92a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: MGC192.168.123.123@tcp: Connection restored to 192.168.123.123@tcp (at 0@lo) Lustre: Skipped 43 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1406 to 0x0:1441 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1389 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1456 to 0x0:1473 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1484 to 0x0:1569 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000-mdc-ffff880262f6d3d8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 36 previous similar messages LustreError: 20814:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802aff1d3d8 ns: mdt-lustre-MDT0000_UUID lock: ffff88007707cf00/0x836aa9adb87aad09 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0xcf:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x836aa9adb87aacfb expref: 4 pid: 20814 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 6 previous similar messages Lustre: 22501:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732403592/real 1732403592] req@ffff88029ed00040 x1816554542697024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1732403635 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:1.0' Lustre: 22501:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 106 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1525 to 0x0:1569 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1568 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1504 to 0x0:1569 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1651 to 0x0:1697 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff880262f6d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages LustreError: 26161:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x637:0x0].0x0 (ffff88029af320c0) refcount nonzero (2) after lock cleanup; forcing cleanup. | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 16292:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 16292:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 16292, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 18611:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802cc044138 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b4e843c0/0x67f684492ff63e76 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x21:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x67f684492ff63e68 expref: 3 pid: 18611 timeout: 0 lvb_type: 0 LustreError: 11-0: lustre-MDT0000-mdc-ffff8802bddf6678: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 36 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 15 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1713 to 0x0:1729 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1649 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1649 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1649 to 0x0:1665 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 15429:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880087aad180 x1816545305427904/t30064771187(30064771187) o101->lustre-MDT0001-mdc-ffff8802bddf6678@0@lo:12/10 lens 648/600 e 0 to 0 dl 1732395222 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 15429:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 62 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:283 to 0x2c0000401:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:219 to 0x380000401:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:219 to 0x340000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:218 to 0x300000401:257 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:323 to 0x380000400:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:323 to 0x300000400:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:387 to 0x2c0000400:417 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:323 to 0x340000400:353 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 17912:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 582, rollback = 2 Lustre: 17912:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/1, destroy: 0/0/0 Lustre: 17912:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 17912:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 17912:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/2, delete: 0/0/0 Lustre: 17912:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1706 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1706 to 0x0:1729 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1770 to 0x0:1793 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1706 to 0x0:1729 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 18518:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 513 < left 582, rollback = 2 Lustre: 18518:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 23 previous similar messages Lustre: 18518:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 Lustre: 18518:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 18518:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 18518:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 18518:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 18518:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 18518:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 18518:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: 18518:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 18518:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 23 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:303 to 0x300000401:321 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:367 to 0x2c0000401:385 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:302 to 0x340000401:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:303 to 0x380000401:321 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 21787:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802a616ca88 ns: mdt-lustre-MDT0002_UUID lock: ffff8802a9e9a200/0x67f684492ffe8edf lrc: 3/0,0 mode: PR/PR res: [0x280000404:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x67f684492ffe8ed1 expref: 13 pid: 21787 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:427 to 0x380000400:449 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:427 to 0x340000400:449 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:492 to 0x2c0000400:513 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:427 to 0x300000400:449 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 35 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 778 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1770 to 0x0:1793 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1770 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1770 to 0x0:1793 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1834 to 0x0:1857 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.163@tcp (at 0@lo) Lustre: Skipped 160 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:413 to 0x2c0000401:449 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:349 to 0x300000401:385 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:349 to 0x340000401:385 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:349 to 0x380000401:385 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times Lustre: lustre-MDT0002-mdc-ffff8802bddf6678: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 131 previous similar messages LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:525 to 0x2c0000400:545 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:464 to 0x340000400:481 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:463 to 0x380000400:481 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:460 to 0x300000400:481 Lustre: DEBUG MARKER: centos-161.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times | Link to test |
| replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) | LustreError: 27461:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27461:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27461, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 11 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 32685:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 32685:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 32685, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 17817:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 17817:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 17817, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88026a1ddd28: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages | Link to test |
| replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update | LustreError: 11622:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 11622:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 11622, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 18861:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff880299f54ad8 x1816495692695744/t107374182411(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:247/0 lens 2200/4320 e 0 to 0 dl 1732347747 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1377 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1155 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1097 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1059 to 0x0:1249 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| recovery-small test 54: back in time | LustreError: 20887:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 20887:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 20887, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 10: resending a replayed unlink | LustreError: 24733:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 24733:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 24733, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| recovery-small test 106: lightweight connection support | LustreError: 4039:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 4039:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 4039, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=805, val=0*** Lustre: Mounted lustre-client LustreError: 15639:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 19: resend of open request | LustreError: 22192:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 22192:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 22192, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 1584:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802af61d180 x1816455040974720/t77309411443(0) o101->9cfc3800-6d13-45b0-ab40-8ae82d544537@0@lo:512/0 lens 664/656 e 0 to 0 dl 1732307997 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client 9cfc3800-6d13-45b0-ab40-8ae82d544537 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8800a8409bf8: Connection restored to (at 0@lo) Lustre: Skipped 22 previous similar messages Lustre: 1589:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802dcb50040 x1816455040974720/t77309411443(0) o101->9cfc3800-6d13-45b0-ab40-8ae82d544537@0@lo:519/0 lens 664/3424 e 0 to 0 dl 1732308004 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 23688:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23688:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23688, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0000-mdc-ffff88009a8b37e8: can't stat MDS #0: rc = -107 LustreError: 7749:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732306572 with bad export cookie 10965402697336026886 LustreError: 7749:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0000-mdc-ffff88009a8b37e8: can't stat MDS #0: rc = -19 LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 5 previous similar messages LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0000-mdc-ffff88009a8b37e8: can't stat MDS #0: rc = -19 LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 2 previous similar messages LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0000-mdc-ffff88009a8b37e8: can't stat MDS #0: rc = -19 LustreError: 4699:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 5 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1614 to 0x0:1633 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1742 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1679 to 0x0:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1679 to 0x0:1697 Lustre: DEBUG MARKER: centos-216.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LustreError: 4608:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d5b4ea80 x1816452156337472/t0(0) o2->lustre-OST0003-osc-MDT0001@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-3-1.0' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 983 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.218@tcp (at 0@lo) Lustre: Skipped 164 previous similar messages Lustre: 5676:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 510 < left 582, rollback = 2 Lustre: 5676:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/2, destroy: 0/0/0 Lustre: 5676:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 5676:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 5676:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/3, delete: 0/0/0 Lustre: 5676:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:230 to 0x380000401:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:229 to 0x340000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:229 to 0x300000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:293 to 0x2c0000401:321 Lustre: DEBUG MARKER: centos-216.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LustreError: 11-0: lustre-MDT0002-mdc-ffff88009a8b37e8: operation mds_reint to node 0@lo failed: rc = -107 LustreError: Skipped 27 previous similar messages Lustre: lustre-MDT0002-mdc-ffff88009a8b37e8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 141 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 27 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 19057:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 19057:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 19057, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: 12487:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732298977/real 1732298977] req@ffff8800ab268cc0 x1816444341495552/t0(0) o400->MGC192.168.123.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1732298984 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:4.0' Lustre: 12487:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages LustreError: 12472:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802d2358040 x1816444341324288/t154618822754(154618822754) o101->lustre-MDT0000-mdc-ffff8802bf4e2548@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732299046 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 12472:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 10 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1644 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1645 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1708 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1612 to 0x0:1633 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8803201637e8: can't stat MDS #0: rc = -107 LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8803201637e8: can't stat MDS #0: rc = -19 LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 1 previous similar message LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8803201637e8: can't stat MDS #0: rc = -19 LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 9 previous similar messages LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8803201637e8: can't stat MDS #0: rc = -19 LustreError: 12573:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 10 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:292 to 0x2c0000401:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:228 to 0x380000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:228 to 0x300000401:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:197 to 0x340000401:225 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:33 to 0x340000400:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:33 to 0x300000400:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:97 to 0x2c0000400:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:33 to 0x380000400:65 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGC192.168.123.153@tcp: Connection restored to 192.168.123.153@tcp (at 0@lo) Lustre: Skipped 170 previous similar messages Lustre: 14810:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 582, rollback = 2 Lustre: 14810:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 6/24/1, destroy: 0/0/0 Lustre: 14810:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 14810:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 11/331/0, punch: 0/0/0, quota 1/3/0 Lustre: 14810:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 18/334/2, delete: 0/0/0 Lustre: 14810:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2084 to 0x0:2113 Lustre: lustre-OST0001: deleting orphan objects from 0x0:2021 to 0x0:2049 Lustre: lustre-OST0003: deleting orphan objects from 0x0:2021 to 0x0:2049 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1989 to 0x0:2017 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 147 previous similar messages LustreError: 14470:0:(lmv_obd.c:1273:lmv_statfs()) lustre-MDT0001-mdc-ffff8803201637e8: can't stat MDS #0: rc = -107 LustreError: 14470:0:(lmv_obd.c:1273:lmv_statfs()) Skipped 7 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1018 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:315 to 0x380000401:353 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:282 to 0x340000401:321 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:379 to 0x2c0000401:417 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:314 to 0x300000401:353 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times LustreError: 11-0: lustre-MDT0002-mdc-ffff8802bf4e2548: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 33 previous similar messages LustreError: 15618:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732299215 with bad export cookie 13395842915010014681 LustreError: 15618:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:119 to 0x340000400:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:184 to 0x2c0000400:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:120 to 0x300000400:161 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:119 to 0x380000400:161 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times LustreError: 166-1: MGC192.168.123.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.153@tcp) after server handle changed from 0xb9e79828301aed0e to 0xb9e79828301f440f Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 14 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: Skipped 14 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 15 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:2101 to 0x0:2145 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2069 to 0x0:2113 Lustre: lustre-OST0001: deleting orphan objects from 0x0:2101 to 0x0:2145 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2165 to 0x0:2209 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: Skipped 15 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: Skipped 15 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:446 to 0x2c0000401:481 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:351 to 0x340000401:385 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:382 to 0x300000401:417 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:383 to 0x380000401:417 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times LustreError: 15627:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732299381 with bad export cookie 13395842915010473076 LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:246 to 0x2c0000400:289 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:185 to 0x380000400:225 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:183 to 0x340000400:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:185 to 0x300000400:225 Lustre: DEBUG MARKER: centos-151.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times | Link to test |
| replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 | LustreError: 24388:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 24388:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 24388, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 30940:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 30: layout lock replay is not blocked on IO | LustreError: 24558:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 24558:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 24558, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 3279:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32e sleeping for 4000ms | Link to test |
| recovery-small test 61: Verify to not reuse orphan objects - bug 17025 | LustreError: 23193:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 23193:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 23193, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 28097:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 28097:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 28097, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802a27a12a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 18309:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008e248680 x1816408976162432/t94489280726(94489280726) o101->lustre-MDT0000-mdc-ffff8802a27a12a8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732264506 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 18309:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1230 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1230 to 0x0:1249 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1230 to 0x0:1249 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1294 to 0x0:1313 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ad6a92a8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1401 to 0x0:1441 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1338 to 0x0:1377 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1337 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1337 to 0x0:1377 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 20675:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732264577 with bad export cookie 15445676598932453060 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages LustreError: 9257:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88028182b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b8de9e40/0xd65a110832f0431d lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x30:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd65a110832f0430f expref: 3 pid: 9257 timeout: 0 lvb_type: 0 LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1484 to 0x0:1505 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1483 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1483 to 0x0:1505 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1548 to 0x0:1569 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802a27a12a8: operation mds_close to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGC192.168.123.193@tcp: Connection restored to 192.168.123.193@tcp (at 0@lo) Lustre: Skipped 49 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1600 to 0x0:1633 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1664 to 0x0:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1600 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1600 to 0x0:1633 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 10732:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880296e92e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b0905680/0xd65a110832f2f108 lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x121:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd65a110832f2f0fa expref: 3 pid: 10732 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: 18312:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732264726/real 1732264726] req@ffff8802f6454b40 x1816408979007168/t0(0) o400->MGC192.168.123.193@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1732264733 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:2.0' Lustre: 18312:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 43 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1730 to 0x0:1761 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1729 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1730 to 0x0:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1794 to 0x0:1825 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1840 to 0x0:1857 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1924 to 0x0:1953 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1873 to 0x0:1889 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1847 to 0x0:1889 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 8 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff8802a27a12a8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802a27a12a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 49 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.123.193@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.193@tcp) after server handle changed from 0xd65a110832f44695 to 0xd65a110832f61663 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x0:1943 to 0x0:1985 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1941 to 0x0:1985 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2008 to 0x0:2049 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: deleting orphan objects from 0x0:2002 to 0x0:2017 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1911 to 0x0:1953 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2067 to 0x0:2113 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2003 to 0x0:2049 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:2136 to 0x0:2177 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2071 to 0x0:2113 Lustre: lustre-OST0001: deleting orphan objects from 0x0:2039 to 0x0:2081 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1975 to 0x0:2017 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times | Link to test |
| replay-single test 33b: test fid seq allocation | LustreError: 3626:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 3626:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 3626, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 13: close resend timeout | LustreError: 465:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 465:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 465, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 | LustreError: 31193:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 31193:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 31193, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 15 previous similar messages [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 23652:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7b9a5c0 x1816379395297600/t137438953491(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:319/0 lens 1488/4320 e 0 to 0 dl 1732236834 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' | Link to test |
| replay-dual test 10: resending a replayed unlink | LustreError: 31972:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 31972:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 31972, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 5287:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5287:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5287, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009add4138: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1295 to 0x0:1313 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1231 to 0x0:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1231 to 0x0:1249 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1231 to 0x0:1249 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009add4138: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1340 to 0x0:1377 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1339 to 0x0:1377 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1339 to 0x0:1377 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1404 to 0x0:1441 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9bcae98: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 4797:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802c558ca88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cb8fed00/0xb0292c8b7a3f10e5 lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x33:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb0292c8b7a3f10d7 expref: 4 pid: 4797 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 13815:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802dec3f700 x1816340293398848/t94489280764(94489280764) o101->lustre-MDT0000-mdc-ffff88009add4138@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732199166 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 13815:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 305 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1484 to 0x0:1505 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1485 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1485 to 0x0:1505 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1549 to 0x0:1569 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a9bcae98: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGC192.168.123.193@tcp: Connection restored to 192.168.123.193@tcp (at 0@lo) Lustre: Skipped 49 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1668 to 0x0:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1604 to 0x0:1633 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1604 to 0x0:1633 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1605 to 0x0:1633 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 6818:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x2000013a1:0x67c:0x0].0x0 (ffff8800a69925c0) refcount nonzero (2) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: 13828:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732199257/real 1732199257] req@ffff8800a9a94b40 x1816340296361088/t0(0) o400->MGC192.168.123.193@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1732199264 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:3.0' Lustre: 13828:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 116 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1803 to 0x0:1825 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1740 to 0x0:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1740 to 0x0:1761 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1739 to 0x0:1761 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009add4138: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages | Link to test |
| replay-dual test 5: open, unlink |X| close | LustreError: 6827:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 6827:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 6827, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 7290:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| recovery-small test 54: back in time | LustreError: 1422:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 1422:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 1422, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: Mounted lustre-client | Link to test |
| replay-dual test 0a: expired recovery with lost client | LustreError: 27952:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27952:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27952, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 3787:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 12450:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 12450:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 12450, comm: kworker/u32:1 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-OST0000: deleting orphan objects from 0x0:1699 to 0x0:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1635 to 0x0:1665 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1636 to 0x0:1665 Lustre: DEBUG MARKER: centos-216.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LustreError: 11-0: lustre-MDT0001-mdc-ffff8800a86d0958: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 27 previous similar messages Lustre: lustre-MDT0001-mdc-ffff8800a86d0958: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 152 previous similar messages LustreError: 2100:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732138282 with bad export cookie 4965587258662874443 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 32 previous similar messages tgt_recover_1 (27587) used greatest stack depth: 9656 bytes left Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:209 to 0x340000401:225 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:209 to 0x380000401:225 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:208 to 0x300000400:225 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:274 to 0x2c0000401:289 Lustre: DEBUG MARKER: centos-216.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times | Link to test |
| replay-dual test 19: resend of open request | LustreError: 5414:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 5414:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 5414, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=157, val=2147483648*** LustreError: 16688:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8800a6ad1940 x1816252630146432/t77309411443(0) o101->708d3c40-6958-4061-9d94-653867039177@0@lo:710/0 lens 664/656 e 0 to 0 dl 1732114915 ref 1 fl Interpret:/0/0 rc 0/0 job:'createmany.0' Lustre: lustre-MDT0000: Client 708d3c40-6958-4061-9d94-653867039177 (at 0@lo) reconnecting Lustre: 16688:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800ab95ea80 x1816252630146432/t77309411443(0) o101->708d3c40-6958-4061-9d94-653867039177@0@lo:717/0 lens 664/3424 e 0 to 0 dl 1732114922 ref 1 fl Interpret:H/2/0 rc 0/0 job:'createmany.0' | Link to test |
| replay-dual test 12: open resend timeout | LustreError: 14519:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 14519:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 14519, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |
| replay-dual test 9: resending a replayed create | LustreError: 30858:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 30858:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 30858, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 2: |X| mkdir adir | LustreError: 11558:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 11558:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 11558, comm: kworker/u32:3 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 2042:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 6050:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 6050:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 6050, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff8802e78412a8: operation mds_close to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: deleting orphan objects from 0x0:1298 to 0x0:1313 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1234 to 0x0:1249 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1234 to 0x0:1249 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1234 to 0x0:1249 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88009331e678: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LustreError: 16286:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f1be4b40 x1816236857435328/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 16274:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a489e440 x1816236856607808/t94489280766(94489280766) o101->lustre-MDT0000-mdc-ffff8802e78412a8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732100408 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 16274:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 161 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x0:1330 to 0x0:1345 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1330 to 0x0:1345 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1331 to 0x0:1377 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1395 to 0x0:1441 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 0a: expired recovery with lost client | LustreError: 30106:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 30106:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 30106, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | LustreError: 13012:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 1537:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 1537:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 1537, comm: kworker/u32:2 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88028f66ae98: operation mds_close to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 13492:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008fe65180 x1816194865377088/t94489280846(94489280846) o101->lustre-MDT0000-mdc-ffff88032405d3d8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732060390 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 13492:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: lustre-OST0001: deleting orphan objects from 0x0:1238 to 0x0:1281 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1237 to 0x0:1281 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1237 to 0x0:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1301 to 0x0:1345 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88028f66ae98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1371 to 0x0:1409 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1370 to 0x0:1409 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1371 to 0x0:1409 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1435 to 0x0:1473 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88032405d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x0:1495 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1496 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1559 to 0x0:1601 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1495 to 0x0:1537 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88032405d3d8: operation mds_close to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGC192.168.123.123@tcp: Connection restored to 192.168.123.123@tcp (at 0@lo) Lustre: Skipped 43 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1626 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1626 to 0x0:1665 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1690 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1625 to 0x0:1665 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: 13495:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1732060501/real 1732060501] req@ffff880094248680 x1816194867381120/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1732060545 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u32:0.0' Lustre: 13495:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 148 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88028f66ae98: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1761 to 0x0:1793 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1760 to 0x0:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1760 to 0x0:1793 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1825 to 0x0:1857 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88032405d3d8: operation mds_reint to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1959 to 0x0:1985 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1895 to 0x0:1921 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1895 to 0x0:1921 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1895 to 0x0:1921 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: Skipped 8 previous similar messages LustreError: 11-0: lustre-MDT0000-mdc-ffff88028f66ae98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ffff88028f66ae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 49 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.123.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: Evicted from MGS (at 192.168.123.123@tcp) after server handle changed from 0x1acf053984685830 to 0x1acf05398469ad31 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x0:1993 to 0x0:2017 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1991 to 0x0:2017 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1987 to 0x0:2017 Lustre: lustre-OST0000: deleting orphan objects from 0x0:2052 to 0x0:2081 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times LustreError: 11-0: lustre-MDT0000-mdc-ffff88032405d3d8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages | Link to test |
| replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir | LustreError: 27855:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 27855:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 27855, comm: kworker/u32:0 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | LustreError: 14833:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 14833:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 14833, comm: kworker/u32:5 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 14 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 17342:0:(client.c:3185:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802ba791300 x1816169048181056/t154618822740(154618822740) o101->lustre-MDT0000-mdc-ffff8803260153d8@0@lo:12/10 lens 576/600 e 0 to 0 dl 1732036623 ref 2 fl Interpret:RPQU/4/0 rc 301/301 job:'dbench.0' LustreError: 17342:0:(client.c:3185:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:1707 to 0x0:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1642 to 0x0:1665 Lustre: lustre-OST0003: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1643 to 0x0:1665 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 18577:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 582, rollback = 2 Lustre: 18577:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/0, destroy: 0/0/0 Lustre: 18577:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 18577:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 18577:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/1, delete: 0/0/0 Lustre: 18577:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:229 to 0x380000401:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000400:230 to 0x300000400:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:229 to 0x340000401:257 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:293 to 0x2c0000401:321 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:28 to 0x340000400:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:28 to 0x380000400:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:92 to 0x2c0000400:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:29 to 0x300000401:65 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 132 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 20 previous similar messages LustreError: 18220:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1732036696 with bad export cookie 6347990306034941484 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 790 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGC192.168.123.193@tcp: Connection restored to 192.168.123.193@tcp (at 0@lo) Lustre: Skipped 152 previous similar messages Lustre: 19860:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 510 < left 582, rollback = 2 Lustre: 19860:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 11 previous similar messages Lustre: 19860:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0 Lustre: 19860:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 11 previous similar messages Lustre: 19860:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 9/582/0 Lustre: 19860:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 11 previous similar messages Lustre: 19860:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5/127/0, punch: 0/0/0, quota 1/3/0 Lustre: 19860:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 11 previous similar messages Lustre: 19860:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 12/235/3, delete: 0/0/0 Lustre: 19860:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 11 previous similar messages Lustre: 19860:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: 19860:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 11 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x0:2081 to 0x0:2113 Lustre: lustre-OST0003: deleting orphan objects from 0x0:2016 to 0x0:2049 Lustre: lustre-OST0001: deleting orphan objects from 0x0:2016 to 0x0:2049 Lustre: lustre-OST0002: deleting orphan objects from 0x0:2016 to 0x0:2049 Lustre: DEBUG MARKER: centos-191.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times | Link to test |
| replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 | LustreError: 14286:0:(tgt_lastrcvd.c:457:tgt_client_free()) ASSERTION( lut && lut->lut_client_bitmap ) failed: LustreError: 14286:0:(tgt_lastrcvd.c:457:tgt_client_free()) LBUG Pid: 14286, comm: kworker/u32:4 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022 Call Trace: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [<0>] lbug_with_loc+0x4c/0xa0 [libcfs] [<0>] tgt_client_free+0x31d/0x390 [ptlrpc] [<0>] mdt_destroy_export+0xc9/0x310 [mdt] [<0>] class_export_destroy+0x100/0x5d0 [obdclass] [<0>] obd_zombie_exp_cull+0x12/0x20 [obdclass] [<0>] process_one_work+0x18d/0x4a0 [<0>] worker_thread+0x126/0x3b0 [<0>] kthread+0xe4/0xf0 [<0>] ret_from_fork_nospec_begin+0x7/0x21 [<0>] 0xfffffffffffffffe | Lustre: *** cfs_fail_loc=1701, val=2147483648*** LustreError: 1263:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@ffff8802a9906a80 x1816156908374400/t137438953491(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:135/0 lens 1488/4320 e 0 to 0 dl 1732024495 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: deleting orphan objects from 0x0:1379 to 0x0:1505 Lustre: lustre-OST0002: deleting orphan objects from 0x0:1411 to 0x0:1537 Lustre: lustre-OST0001: deleting orphan objects from 0x0:1475 to 0x0:1537 Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1569 Lustre: DEBUG MARKER: centos-226.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 | Link to test |