Edit new crash report

Editing crashreport #70375

Reason	Crashing Function	Where to cut Backtrace	Reports Count
ASSERTION( data != ((void *)0) ) failed	ldlm_server_completion_ast	ldlm_server_completion_ast cleanup_resource ldlm_resource_clean cfs_hash_for_each_relax cfs_hash_for_each_nolock ldlm_namespace_cleanup __ldlm_namespace_free ldlm_namespace_free_prior mdt_device_fini obd_precleanup class_cleanup class_process_config class_manual_cleanup server_put_super generic_shutdown_super kill_anon_super lustre_kill_super deactivate_locked_super deactivate_super cleanup_mnt __cleanup_mnt task_work_run do_notify_resume int_signal	180

Reason

Crashing Function

Where to cut Backtrace

Reports Count

ASSERTION( data != ((void *)0) ) failed

ldlm_server_completion_ast

ldlm_server_completion_ast
cleanup_resource
ldlm_resource_clean
cfs_hash_for_each_relax
cfs_hash_for_each_nolock
ldlm_namespace_cleanup
__ldlm_namespace_free
ldlm_namespace_free_prior
mdt_device_fini
obd_precleanup
class_cleanup
class_process_config
class_manual_cleanup
server_put_super
generic_shutdown_super
kill_anon_super
lustre_kill_super
deactivate_locked_super
deactivate_super
cleanup_mnt
__cleanup_mnt
task_work_run
do_notify_resume
int_signal

180

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):

Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):

Limit to a test:
(Copy from below "Failing text"):

Delete these reports as invalid (real bug in review or some such)

Bug or comment:

Extra info:

Failures list (last 100):

Failing Test	Full Crash	Messages before crash	Comment
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 25492:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25492:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 25492 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa016eb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa069b229>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0677eb7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067802e>] ldlm_resource_clean+0x3e/0x50 [ptlrpc] [<ffffffffa0380ddb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0677ff0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0677ff0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0383f96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0676650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06770e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06776bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12d2b5c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa0392d53>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa037d919>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0398e65>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03999c7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa039bbcc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa039bcb0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0726234>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15f311b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=24171 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8803265f1bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 55 previous similar messages LustreError: 19879:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802aa25a548 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a8aff0c0/0xfa74e1f72e8d207f lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf1b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xfa74e1f72e8d2071 expref: 2 pid: 19879 timeout: 0 lvb_type: 0	Link to test
replay-dual test 26: dbench and tar with mds failover	LustreError: 5059:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 5059:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 5059 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018fb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0693229>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa066feb7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067002e>] ldlm_resource_clean+0x3e/0x50 [ptlrpc] [<ffffffffa0378ddb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa066fff0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa066fff0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa037bf96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa066e650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa066f0e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffffa066f6bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12cdb5c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa038ad53>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa0375a74>] ? class_disconnect_exports+0x274/0x310 [obdclass] [<ffffffffa0390e65>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03919c7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa0393cb0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa071e234>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15ee11b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1245 to 0x280000400:1281) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1245 to 0x2c0000400:1281) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1246 to 0x240000400:1281) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1244 to 0x300000400:1281) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 4814:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 4814:0:(obd_class.h:479:obd_check_dev()) Skipped 95 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 11562:0:(client.c:3395:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88029cbc4b40 x1835105036820864/t94489280689(94489280689) o101->lustre-MDT0000-mdc-ffff8802ae61d3d8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1750094129 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 11562:0:(client.c:3395:ptlrpc_replay_interpret()) Skipped 103 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1350 to 0x2c0000400:1377) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1343 to 0x280000400:1377) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1343 to 0x300000400:1377) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1332 to 0x240000400:1377) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1461 to 0x300000400:1505) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1461 to 0x280000400:1505) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1461 to 0x240000400:1505) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1461 to 0x2c0000400:1505) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1584 to 0x240000400:1601) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1585 to 0x2c0000400:1601) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1585 to 0x280000400:1601) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1585 to 0x300000400:1601) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1674 to 0x2c0000400:1697) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1671 to 0x300000400:1697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1678 to 0x240000400:1697) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1673 to 0x280000400:1697) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 17194:0:(osd_handler.c:699:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 17194:0:(osd_handler.c:699:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 57 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1790 to 0x300000400:1825) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1796 to 0x240000400:1825) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1791 to 0x280000400:1825) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1789 to 0x2c0000400:1825) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.83@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1913 to 0x240000400:1953) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1890 to 0x300000400:1921) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1900 to 0x2c0000400:1921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1903 to 0x280000400:1921) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2000 to 0x240000400:2017) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1969 to 0x2c0000400:1985) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1969 to 0x280000400:1985) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1968 to 0x300000400:1985) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2001 to 0x300000400:2017) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2033 to 0x240000400:2049) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2002 to 0x2c0000400:2017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2001 to 0x280000400:2017) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2065 to 0x240000400:2081) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2030 to 0x300000400:2049) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2029 to 0x280000400:2049) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2025 to 0x2c0000400:2049) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 11 times Lustre: Failing over lustre-MDT0000 LustreError: 32200:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2102 to 0x240000400:2145) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2069 to 0x300000400:2113) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2069 to 0x280000400:2113) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2069 to 0x2c0000400:2113) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: 3988:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f1df92a8 ns: mdt-lustre-MDT0000_UUID lock: ffff880291f1b880/0xe92c5daaa921f7b4 lrc: 4/0,0 mode: CW/CW res: [0x20000afe1:0x214:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xe92c5daaa921f78a expref: 3 pid: 3988 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 20648:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 20648:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 20648 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa017cb1d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa068c1c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0668e77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0668fdf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0371dcb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0668fb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0668fb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0374f86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0667610>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06680a1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa066867b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12cab5c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa0383d43>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa036e909>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0389e55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa038a9b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa038cbbc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa038cca0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07170e4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15eaeeb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=10442 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.38@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3915 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 13317:0:(osd_handler.c:699:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 13317:0:(osd_handler.c:699:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 13729:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 13729:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.38@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3951 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4047 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3952 to 0x300000400:3969) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800a6740008: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4083 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3987 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3988 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4045 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4045 to 0x300000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4045 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4140 to 0x240000400:4161) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 19071:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ad39dd28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cb3596c0/0x2cc68a64d2d3cd2 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1012:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2cc68a64d2d3cc4 expref: 4 pid: 19071 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 26204:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25834:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ae598958 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a97a5a40/0x284e9d3beaa92505 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1287:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x284e9d3beaa924f7 expref: 3 pid: 25834 timeout: 0 lvb_type: 0 LustreError: 26204:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 8 PID: 26204 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a8b1d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d01c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa06ace77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06acfdf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03b5dcb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06acfb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06acfb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03b8f86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06ab610>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06ac0a1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06ac67b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f46b5c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03c7d43>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03b2909>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03cde55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ce9b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d0bbc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03d0ca0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa075b0e4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1789eeb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=21468 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 19263:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8803222c6678 ns: mdt-lustre-MDT0000_UUID lock: ffff88031917c000/0x284e9d3beaa210f3 lrc: 3/0,0 mode: --/CW res: [0x20001a9e3:0xf44:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x284e9d3beaa210e5 expref: 4 pid: 19263 timeout: 0 lvb_type: 0 LustreError: 11829:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802eb020a40 x1834419783484032/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 LustreError: 11829:0:(client.c:1375:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3924 to 0x240000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3969 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3924 to 0x2c0000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3925 to 0x300000400:3969) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3997 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3997 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3997 to 0x2c0000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8803217d6678: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages LustreError: 22423:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a1269bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e2d6b4c0/0x284e9d3beaa3c9c2 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xef7:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x284e9d3beaa3c9b4 expref: 4 pid: 22423 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 22797:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 22797:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.108@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4061 to 0x240000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4062 to 0x2c0000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4093 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4062 to 0x300000400:4097) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8803217d6678: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4125 to 0x240000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4157 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4125 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4126 to 0x2c0000400:4161) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4217 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4184 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4184 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4184 to 0x300000400:4225) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 29911:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749441710/real 1749441710] req@ffff8802f9ffa340 x1834419786363776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1749441726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 29911:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 42 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4252 to 0x300000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4253 to 0x240000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4285 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4253 to 0x2c0000400:4289) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4348 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4317 to 0x2c0000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4317 to 0x240000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4316 to 0x300000400:4353) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4410 to 0x280000400:4449) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4378 to 0x300000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4378 to 0x240000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4378 to 0x2c0000400:4417) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18982:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 18711:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a0b36678 ns: mdt-lustre-MDT0000_UUID lock: ffff8800b4ade580/0x440459d69d01c75b lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x440459d69d01c74d expref: 3 pid: 18711 timeout: 0 lvb_type: 0 LustreError: 18982:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 18982 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a0afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d9139>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06b5de7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06b5f4f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03bea5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06b5f20>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06b5f20>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03c1c16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06b4580>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06b5011>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06b55eb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f34b2c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03d09d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03bb599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03d6ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03d7647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d984c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03d9930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07643f4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa17764ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=14815 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4030 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3965 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3966 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3966 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802ee0f5d28: operation mds_close to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.93@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4094 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4030 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4030 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4030 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802ee0f5d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages LustreError: 16707:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 16707:0:(obd_class.h:479:obd_check_dev()) Skipped 73 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4102 to 0x280000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4166 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4101 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4101 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 17009:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 17009:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4163 to 0x2c0000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4163 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4226 to 0x240000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4163 to 0x300000400:4193) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.93@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4224 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4289 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4225 to 0x300000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4224 to 0x280000400:4257) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 23161:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1749196903/real 1749196903] req@ffff8802ea80e940 x1834163058926464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1749196919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 23161:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 63 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4289 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4289 to 0x2c0000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4352 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4289 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 12195:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 11821:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880321f6d3d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f7694b40/0x2e3fb03cbb85516 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x13fc:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2e3fb03cbb85508 expref: 4 pid: 11821 timeout: 0 lvb_type: 0 LustreError: 12195:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 11 PID: 12195 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a6afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06e4139>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06c0a47>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06c0bac>] ldlm_resource_clean+0x2c/0x50 [ptlrpc] [<ffffffffa03c9a5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06c0b80>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06c0b80>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03ccc16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06bf4a0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06c00d1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06c06ab>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f13b2c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03db9d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03c6599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03e1ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03e2647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03e484c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03e4930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa076f3f4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa17884ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5758 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 3540:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f71b4a88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f389cf00/0x2e3fb03cbae0dfd lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xedb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50386400000000 nid: 0@lo remote: 0x2e3fb03cbae0def expref: 3 pid: 3540 timeout: 0 lvb_type: 0 LustreError: 16624:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a4cd1440 x1833365331217024/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 LustreError: 16624:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3924 to 0x240000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3924 to 0x2c0000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3925 to 0x300000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3969 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4000 to 0x2c0000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3999 to 0x240000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4031 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3999 to 0x300000400:4033) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 28489:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748436019 with bad export cookie 208286001509897089 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4062 to 0x300000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4094 to 0x280000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4062 to 0x240000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4062 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800ac76c138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 36 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection restored to 192.168.123.13@tcp (at 0@lo) Lustre: Skipped 33 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4127 to 0x240000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4158 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4126 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4126 to 0x2c0000400:4161) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 8217:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 8217:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4223 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4191 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4191 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4190 to 0x2c0000400:4225) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 14112:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880325faf340 x1833365335157504/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.13@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4255 to 0x2c0000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4255 to 0x300000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4286 to 0x280000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4254 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 14114:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748436103/real 1748436103] req@ffff8802eb8d3740 x1833365335165184/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748436119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 14114:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 64 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4343 to 0x280000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4312 to 0x300000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4312 to 0x240000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4312 to 0x2c0000400:4353) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4380 to 0x300000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4380 to 0x2c0000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4380 to 0x240000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4412 to 0x280000400:4449) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4441 to 0x2c0000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4441 to 0x300000400:4481) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4473 to 0x280000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4441 to 0x240000400:4481) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 14108:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a4cd4640 x1833365330579328/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8800ac76c138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1748436229 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 14108:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 483 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4512 to 0x300000400:4545) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4544 to 0x280000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4511 to 0x240000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4512 to 0x2c0000400:4545) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4575 to 0x240000400:4609) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4575 to 0x300000400:4609) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4575 to 0x2c0000400:4609) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4607 to 0x280000400:4641) Lustre: DEBUG MARKER: centos-11.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 30945:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 29498:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880092acc138 ns: mdt-lustre-MDT0000_UUID lock: ffff8802bc8ded00/0xf7d37fae3ac8ac8a lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xfd0:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xf7d37fae3ac8ac7c expref: 3 pid: 29498 timeout: 0 lvb_type: 0 LustreError: 30945:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 8 PID: 30945 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0180afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a5089>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0681d37>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0681e9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa038aa5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0681e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0681e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa038dc16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06804d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0680f61>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068153b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12a8a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa039c9d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa0387599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03a2ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03a3647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a584c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03a5930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa072fa54>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15c849b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=22228 ... LustreError: 23203:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 23203:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802dc1c4138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 55 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 23629:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 23629:0:(obd_class.h:479:obd_check_dev()) Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 57 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3915 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: 10475:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748329814/real 1748329814] req@ffff8803246da840 x1833253797909888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748329830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 10475:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 59 previous similar messages Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4018 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3955 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3955 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3955 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802dc1c4138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4018 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4019 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4051 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 27464:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 27464:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 6 PID: 27464 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01c7afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06c6089>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06a2d37>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06a2e9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03aba5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06a2e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06a2e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03aec16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06a14d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a1f61>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a253b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f32a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03bd9d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03a8599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c3ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03c4647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03c684c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03c6930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0750a54>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa177549b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=22727 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 20993:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800abb0efc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800aba34780/0x43909ea058e02f28 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x43909ea058e02f1a expref: 3 pid: 20993 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3919 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.18@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 29 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection restored to 192.168.123.18@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031edae678: operation mds_hsm_state_set to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 24051:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 24051:0:(obd_class.h:479:obd_check_dev()) Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4029 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4028 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4092 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4096 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4095 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4159 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4096 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 31148:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a8749440 x1833197183316480/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 31148:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4215 to 0x240000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4152 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4151 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4152 to 0x280000400:4193) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4234 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4298 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4235 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4235 to 0x280000400:4257) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: 31154:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748275819/real 1748275819] req@ffff8802eab8d040 x1833197184304768/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748275835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 31154:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 72 previous similar messages Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4277 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4276 to 0x2c0000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4276 to 0x280000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4341 to 0x240000400:4385) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4351 to 0x2c0000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4415 to 0x240000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4350 to 0x280000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4351 to 0x300000400:4385) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 27093:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e3cedd28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802df1b6940/0x43909ea058e7e1fc lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x43909ea058e7e1ee expref: 3 pid: 27093 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 13591:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 13217:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800bb912548 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f85516c0/0x1a1773d85e9ab52c lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x1a1773d85e9ab51e expref: 3 pid: 13217 timeout: 0 lvb_type: 0 LustreError: 13591:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 13591 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01acafd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06bc089>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0698d37>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0698e9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03a1a5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0698e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0698e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a4c16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06974d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0697f61>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa069853b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f40a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03b39d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa039e599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b9ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ba647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03bc84c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03bc930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0746a54>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa178349b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=7154 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3921 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3966 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3922 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3966 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4090 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4027 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4027 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4027 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031e004138: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.18@tcp (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4160 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4097 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4097 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4097 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88031e004138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 39 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 9236:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f15753d8 ns: mdt-lustre-MDT0000_UUID lock: ffff880325a816c0/0x1a1773d85e93d8c6 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x1112:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1a1773d85e93d8b8 expref: 4 pid: 9236 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 9610:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 9610:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4163 to 0x2c0000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4227 to 0x240000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4164 to 0x300000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4163 to 0x280000400:4193) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.18@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4288 to 0x240000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4224 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4225 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4224 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 15549:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748094477/real 1748094477] req@ffff8802f82da840 x1833007084798848/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1748094493 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 15549:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 63 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4287 to 0x2c0000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4351 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4288 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4287 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4341 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4404 to 0x240000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4341 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4341 to 0x300000400:4385) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4471 to 0x240000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4407 to 0x300000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4406 to 0x2c0000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4407 to 0x280000400:4449) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 32104:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748094575 with bad export cookie 1880098742587592881 LustreError: 12324:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f2798958 ns: mdt-lustre-MDT0000_UUID lock: ffff880323370400/0x1a1773d85e9878bb lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x131a:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1a1773d85e9878ad expref: 4 pid: 12324 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 15540:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802e698e440 x1833007080212480/t317827580345(317827580345) o101->lustre-MDT0000-mdc-ffff88031e004138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1748094618 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 15540:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 553 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4480 to 0x2c0000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4480 to 0x300000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4479 to 0x280000400:4513) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4544 to 0x240000400:4577) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4546 to 0x2c0000400:4577) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4610 to 0x240000400:4641) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4546 to 0x300000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4545 to 0x280000400:4577) Lustre: DEBUG MARKER: centos-16.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 24698:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 24698:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 24698 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0167afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06b3089>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa068fd37>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa068fe9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0398a5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa068fe70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa068fe70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa039bc16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa068e4d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa068ef61>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068f53b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12afa7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03aa9d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa0395599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b0ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03b1647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03b384c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03b3930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa073da54>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15ad49b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=20896 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802dc5b12a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.53@tcp (at 0@lo) Lustre: Skipped 67 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3915 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 24567:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 24567:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 25310:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 25310:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: 3693:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747992639/real 1747992639] req@ffff8802dc775a40 x1832900211908736/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747992655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 3693:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 41 previous similar messages LustreError: MGC192.168.123.53@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3956 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3956 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3956 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4020 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4019 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4083 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4020 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4090 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4090 to 0x300000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4154 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802dc5b12a8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4143 to 0x280000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4143 to 0x300000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4207 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4144 to 0x2c0000400:4161) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4250 to 0x240000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4186 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4186 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4186 to 0x280000400:4225) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4249 to 0x300000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4249 to 0x2c0000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4249 to 0x280000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4313 to 0x240000400:4353) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4364 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4300 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4321) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4345 to 0x300000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4345 to 0x2c0000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4408 to 0x240000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4345 to 0x280000400:4385) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 3682:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802cb4d9e40 x1832900211666688/t317827580345(317827580345) o101->lustre-MDT0000-mdc-ffff8802dc5b12a8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1747992878 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 3682:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 415 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4460 to 0x240000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4396 to 0x300000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4395 to 0x280000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4396 to 0x2c0000400:4417) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4440 to 0x300000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4441 to 0x2c0000400:4481) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4441 to 0x280000400:4481) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4504 to 0x240000400:4545) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4494 to 0x300000400:4513) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4494 to 0x2c0000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4495 to 0x280000400:4513) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4558 to 0x240000400:4577) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 13 times Lustre: Failing over lustre-MDT0000 LustreError: 22165:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a5510958 ns: mdt-lustre-MDT0000_UUID lock: ffff880326ec2d40/0x28f3cdad489d00ed lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xed6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x28f3cdad489d00df expref: 3 pid: 22165 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 26222:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25850:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ac230008 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e6ae0040/0x7964d2cb25c4240 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x11d8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x7964d2cb25c4232 expref: 3 pid: 25850 timeout: 0 lvb_type: 0 LustreError: 26222:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 26222 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a4afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06cb089>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06a7d37>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06a7e9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03b0a5b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06a7e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06a7e70>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03b3c16>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06a64d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a6f61>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a753b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f25a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03c29d3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03ad599>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c8ae5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03c9647>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03cb84c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03cb930>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0755a54>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa176849b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=22622 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802f7a1ca88: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages LustreError: 20417:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802dfbedd28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f56dcf00/0x7964d2cb2566afc lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf04:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x7964d2cb2566aee expref: 4 pid: 20417 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.118@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3919 to 0x240000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3964 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4033 to 0x280000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3969 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3969 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3969 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 31020:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ea35cb40 x1832836989796608/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 LustreError: 31020:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4032 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4032 to 0x240000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4096 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4031 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 31022:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800aa73e440 x1832836990574080/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 LustreError: 31022:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 1 previous similar message LustreError: 24516:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 24516:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4158 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4094 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4094 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4094 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4158 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4158 to 0x300000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4222 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4159 to 0x2c0000400:4193) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4224 to 0x240000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4288 to 0x280000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4224 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4224 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 4865:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 4493:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ef0ad3d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d78d0f40/0xdef1d00d93fd2a3b lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xdef1d00d93fd2a2d expref: 3 pid: 4493 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 4865:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 4 PID: 4865 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01b2afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06f00b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa06ccd67>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06ccecf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03d59ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06ccea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06ccea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03d8b66>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06cb500>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06cbf91>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06cc56b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f06a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03e7923>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03d24e9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03eda35>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ee597>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03f079c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03f0880>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa077a614>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa176b9fb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=30840 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3921 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800b4da2548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 28 previous similar messages LustreError: 31221:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88032624efc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f05ae940/0xdef1d00d93f380b0 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf00:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xdef1d00d93f380a2 expref: 4 pid: 31221 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800b4da2548: operation mds_readpage to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 32165:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 32165:0:(obd_class.h:479:obd_check_dev()) Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4028 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4028 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4027 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4092 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4091 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4092 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4156 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 6875:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747427409/real 1747427409] req@ffff880324060f40 x1832307591215488/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747427425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 6875:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 56 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4153 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4153 to 0x2c0000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4153 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4217 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4232 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4232 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4297 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4232 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4280 to 0x2c0000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4343 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4279 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4280 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4409 to 0x240000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4345 to 0x280000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4345 to 0x300000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4345 to 0x2c0000400:4385) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 6862:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802f62f4640 x1832307587786880/t317827580346(317827580346) o101->lustre-MDT0000-mdc-ffff8800b4da2548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1747427545 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 6862:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 481 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4417 to 0x280000400:4449) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4416 to 0x300000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4416 to 0x2c0000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4480 to 0x240000400:4513) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4543 to 0x240000400:4577) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4479 to 0x300000400:4513) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4479 to 0x2c0000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4479 to 0x280000400:4513) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4608 to 0x240000400:4641) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4544 to 0x2c0000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4543 to 0x280000400:4577) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4544 to 0x300000400:4577) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: 9624:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1747427591 with bad export cookie 16064850102543192474	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 5868:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 9284:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012c0e7000 ns: mdt-lustre-MDT0000_UUID lock: ffff88009d1be600/0x576cf9799fe77303 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1a7a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.29@tcp remote: 0xe754ce5738b21323 expref: 3 pid: 9284 timeout: 0 lvb_type: 0 LustreError: 5868:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 5868 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c3afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa08300b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa080cd67>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa080cecf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa04b38db>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa080cea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa080cea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa04b6a96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa080b500>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa080bf91>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa080c56b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f27acc>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa04c5853>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa04b0419>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa04cb965>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04cc4c7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa04ce6cc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa04ce7b0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08ba614>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1223d6b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=14718 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 17922:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009d293000 ns: mdt-lustre-MDT0000_UUID lock: ffff880092a90c00/0x576cf9799fd0b388 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xc41:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.204.29@tcp remote: 0xe754ce5738ab0228 expref: 4 pid: 17922 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 16968:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup LustreError: 16968:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 13174:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 13174:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 118 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5099 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5135 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19921:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5527 to 0x280000401:5569) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5495 to 0x2c0000401:5537) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24188:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5880 to 0x2c0000401:5921) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5912 to 0x280000401:5953) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28438:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (28438) used greatest stack depth: 9728 bytes left Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:513) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:513) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 13174:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88007f750800 ns: mdt-lustre-MDT0000_UUID lock: ffff880092a8f000/0x576cf9799fde571c lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xbc0:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.29@tcp remote: 0xe754ce5738af3dd8 expref: 4 pid: 13174 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (not set up) Lustre: Skipped 4 previous similar messages Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6294 to 0x2c0000401:6337) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6327 to 0x280000401:6369) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32694:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:545) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:545) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6742 to 0x280000401:6785) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6710 to 0x2c0000401:6753) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4553:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:577) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:577) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 4134:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3761:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880323920958 ns: mdt-lustre-MDT0000_UUID lock: ffff8803216d2200/0xc810621fba2b1792 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x138a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xc810621fba2b1784 expref: 3 pid: 3761 timeout: 0 lvb_type: 0 LustreError: 4134:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 11 PID: 4134 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa019fafd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06e80b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06c4d67>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06c4ecf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03cd9ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06c4ea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06c4ea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03d0b66>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06c3500>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06c3f91>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06c456b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f84a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03df923>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03ca4e9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03e5a35>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03e6597>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03e879c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03e8880>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0772614>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa177b9fb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=30679 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 28476:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 28476:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 9111:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1747284525 with bad export cookie 14416130295619231425 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 31057:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b0abefc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800aac91a80/0xc810621fba228dd9 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xc810621fba228dcb expref: 4 pid: 31057 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-mdc-ffff8802d4fe5d28: Connection restored to 192.168.123.68@tcp (at 0@lo) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4032 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: Skipped 28 previous similar messages Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802d4fe5d28: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802d4fe5d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 29 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 32004:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup LustreError: 32004:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4032 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4096 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4032 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4032 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 6641:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88032015b240 x1832157864429184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 6641:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4157 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4093 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4092 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4093 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.68@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4160 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4159 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4159 to 0x2c0000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4224 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 889:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f189ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f2f28b80/0xc810621fba265e83 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x115b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xc810621fba265e75 expref: 3 pid: 889 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: 6646:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747284622/real 1747284622] req@ffff8802ea350f40 x1832157865234048/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747284638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 6646:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4216 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4215 to 0x300000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4280 to 0x240000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4215 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4281 to 0x280000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4281 to 0x300000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4346 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4281 to 0x2c0000400:4321) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: 6646:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ea03f840 x1832157867561216/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 6646:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4348 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4349 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4348 to 0x2c0000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4412 to 0x240000400:4449) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 6636:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802f7cd3240 x1832157861643520/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8802d4fe5d28@0@lo:12/10 lens 576/608 e 0 to 0 dl 1747284753 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 6636:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 431 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4416 to 0x2c0000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4416 to 0x280000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4480 to 0x240000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4416 to 0x300000400:4449) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4479 to 0x2c0000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4479 to 0x280000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4479 to 0x300000400:4513) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4543 to 0x240000400:4577) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 1152:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 31044:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a99f9bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff880097a28400/0x10e82cc093dc0f57 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x118a:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x10e82cc093dc0f49 expref: 3 pid: 31044 timeout: 0 lvb_type: 0 LustreError: 1152:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 15 PID: 1152 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0189afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06bb0b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06979c7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0697b2c>] ldlm_resource_clean+0x2c/0x50 [ptlrpc] [<ffffffffa03a0ccb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0697b00>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0697b00>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a3e86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0696420>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0697051>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa069762b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12cba7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03b2c43>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa039d809>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b8d55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03b98b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03bbabc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03bbba0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0745614>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15e5d1b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=7602 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3960 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3916 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3916 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3950 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3952 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4014 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 13554:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 13554:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802ec7f0958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages LustreError: 14087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 14087:0:(obd_class.h:479:obd_check_dev()) Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3991 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3987 to 0x2c0000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3981 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4051 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4047 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4111 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4047 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4015 to 0x280000400:4097) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4119 to 0x280000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4150 to 0x300000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4215 to 0x240000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4150 to 0x2c0000400:4193) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4214 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4269 to 0x240000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4180 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4211 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 27385:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1747169152/real 1747169152] req@ffff8802e9458540 x1832036750558976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1747169168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 27385:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 78 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4275 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4272 to 0x2c0000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4281 to 0x300000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4305 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802ec7f0958: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages LustreError: 25806:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802eb586fc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f9855e00/0x10e82cc093dae4fe lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xef5:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x10e82cc093dae4f0 expref: 4 pid: 25806 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4345 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4346 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4320 to 0x2c0000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4337 to 0x240000400:4353) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 27371:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88027fa78540 x1832036747935232/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8802ec7f0958@0@lo:12/10 lens 576/608 e 0 to 0 dl 1747169273 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 27371:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 308 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4394 to 0x280000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4362 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4366 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4393 to 0x300000400:4417) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 25581:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 8754:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88006eef3800 ns: mdt-lustre-MDT0000_UUID lock: ffff88012cc11c00/0x30d408a53442e884 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xbb1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.22@tcp remote: 0x7bc42b3940db7ab7 expref: 3 pid: 8754 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) LustreError: 25581:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 25581 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01a2afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa08c00b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa089cd67>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa089cecf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa05a58db>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa089cea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa089cea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa05a8a96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa089b500>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa089bf91>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa089c56b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f25a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa05b7853>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa05a2419>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05bd965>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05be4c7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05c06cc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa05c07b0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa094aa24>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1220d6b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13542 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) LustreError: 17045:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup LustreError: 17045:0:(obd_class.h:479:obd_check_dev()) Skipped 59 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 8754:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 8754:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 123 previous similar messages Lustre: Skipped 30 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 37 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5098 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5133 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19995:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.122@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5455 to 0x280000401:5505) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5424 to 0x2c0000401:5441) Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24265:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.122@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:481) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 9207:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7662:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f8dc1bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a57ab4c0/0x92ec7a6b1d200acc lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf99:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x92ec7a6b1d200abe expref: 2 pid: 7662 timeout: 0 lvb_type: 0 LustreError: 9207:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 5 PID: 9207 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa019aafd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06be0b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa069a9c7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa069ab2c>] ldlm_resource_clean+0x2c/0x50 [ptlrpc] [<ffffffffa03a3ccb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa069ab00>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa069ab00>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a6e86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0699420>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa069a051>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa069a62b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12a2a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03b5c43>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03a0809>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03bbd55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03bc8b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03beabc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03beba0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0748614>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa159ad1b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=3183 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800aaff8958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 52 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.63@tcp (at 0@lo) Lustre: Skipped 68 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3913 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 6353:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 6353:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 6744:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 6744:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: 25432:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746694303/real 1746694303] req@ffff8802b5a0f340 x1831538845627904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746694319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 25432:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 43 previous similar messages LustreError: MGC192.168.123.63@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3952 to 0x2c0000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4048 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3952 to 0x300000400:3969) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800aaff8958: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 26605:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 26231:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88030fbc0958 ns: mdt-lustre-MDT0000_UUID lock: ffff88031ca32d40/0x95a1ee45869699df lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf06:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x95a1ee45869699d1 expref: 3 pid: 26231 timeout: 0 lvb_type: 0 LustreError: 26605:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 26605 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01daafd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06cc0b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa06a8d67>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06a8ecf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03b18db>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06a8ea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06a8ea0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03b4a96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06a7500>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a7f91>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a856b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f2ba7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03c3853>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03ae419>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c9965>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ca4c7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03cc6cc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03cc7b0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0756a24>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa176ad1b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=25309 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 2417:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800ad346940 x1831487684806400/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 LustreError: 23098:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b5768958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f5752980/0x95a1ee45869499a4 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf07:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x95a1ee4586949996 expref: 4 pid: 23098 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.53@tcp (at 0@lo) Lustre: Skipped 51 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3919 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3968 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4032 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3968 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800ab844a88: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)	Link to test
replay-dual test 26: dbench and tar with mds failover	LustreError: 16846:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 16846:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 16846 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa017cb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d50c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa06b1d77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06b1edf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03bb9cb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06b1eb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06b1eb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03beb86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06b0510>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06b0fa1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffffa06b157b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12a0a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03cd9a3>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa03b8664>] ? class_disconnect_exports+0x274/0x310 [obdclass] [<ffffffffa03d3ab5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03d4617>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d6900>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa075fac4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15beacb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	LustreError: 5516:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5516:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802e0a02548: operation mds_reint to node 0@lo failed: rc = -19 LustreError: 6359:0:(ldlm_resource.c:982:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8802bf5202c0) refcount nonzero (2) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.3@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 3 previous similar messages Lustre: 7406:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 7406:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 2 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1146 to 0x240000400:1185) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1305 to 0x2c0000400:1345) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1176 to 0x280000400:1217) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1304 to 0x300000400:1345) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802e0a02548: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: 10601:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 10601:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 442 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1415 to 0x300000400:1441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1254 to 0x240000400:1281) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1287 to 0x280000400:1313) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1414 to 0x2c0000400:1441) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 13996:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 13996:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 422 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1408 to 0x280000400:1441) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1535 to 0x2c0000400:1569) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1535 to 0x300000400:1569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1375 to 0x240000400:1409) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f288c138: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 2 previous similar messages LustreError: 13611:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e618b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a88b34c0/0xf3cec49c69d165cc lrc: 4/0,0 mode: PR/PR res: [0x20000afe1:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xf3cec49c69d165be expref: 2 pid: 13611 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 32451:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 32451:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 32451 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0181afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa069e0c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa067ad77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067aeee>] ldlm_resource_clean+0x3e/0x50 [ptlrpc] [<ffffffffa03839cb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa067aeb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa067aeb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0386b86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0679510>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0679fa1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa067a57b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12b6a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa0395943>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa0380509>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa039ba55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa039c5b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa039e7bc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa039e8a0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07287e4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15d3aeb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=19524 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff880097982548: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000-mdc-ffff880097982548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 55 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 63 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3960 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3916 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3916 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 9765:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b50b3740 x1830581692549376/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 projid:4294967295 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4023 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3959 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3959 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3959 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 26681:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 26681:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4081 to 0x240000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4017 to 0x2c0000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4016 to 0x280000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4017 to 0x300000400:4033) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 28294:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802c9a76678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802dd54e580/0xc5464951c0d54ad9 lrc: 3/0,0 mode: --/CW res: [0x20001a9e3:0xede:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xc5464951c0d54acb expref: 4 pid: 28294 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 29549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 29549:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages LustreError: 29137:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 29137:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4120 to 0x240000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4121 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4056 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4056 to 0x280000400:4097) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 30380:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009800dd28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802883ccb40/0xc5464951c0d5d41d lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf0a:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xc5464951c0d5d40f expref: 3 pid: 30380 timeout: 0 lvb_type: 0	Link to test
replay-dual test 26: dbench and tar with mds failover	LustreError: 28929:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25566:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800adde12a8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e8081300/0x699aaef116d42256 lrc: 4/0,0 mode: CW/CW res: [0x20000afe1:0x47d:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x699aaef116d4223a expref: 3 pid: 25566 timeout: 0 lvb_type: 0 LustreError: 28929:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 28929 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a3afd>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a10c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa067dd77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067dedf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03869cb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa067deb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa067deb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0389b86>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa067c510>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa067cfa1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffffa067d57b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12c4a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa0398943>] obd_precleanup+0x163/0x290 [obdclass] [<ffffffffa0383664>] ? class_disconnect_exports+0x274/0x310 [obdclass] [<ffffffffa039ea55>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa039f5b7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a18a0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa072b7e4>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15bfaeb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	LustreError: 2823:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 2823:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802b3975d28: operation ldlm_enqueue to node 0@lo failed: rc = -19 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 3 previous similar messages LustreError: 11810:0:(client.c:3393:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880324542840 x1830537580327936/t94489280682(94489280682) o101->lustre-MDT0000-mdc-ffff8802b3975d28@0@lo:12/10 lens 576/608 e 0 to 0 dl 1745738526 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'dbench.0' uid:0 gid:0 projid:0 LustreError: 11810:0:(client.c:3393:ptlrpc_replay_interpret()) Skipped 1 previous similar message Lustre: 4741:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 4741:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 2 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1236 to 0x280000400:1281) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1237 to 0x2c0000400:1281) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1236 to 0x300000400:1281) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1237 to 0x240000400:1281) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802aaf54138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: 8550:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 8550:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 367 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1367 to 0x280000400:1409) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1367 to 0x300000400:1409) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1367 to 0x2c0000400:1409) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1366 to 0x240000400:1409) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 12381:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 12381:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 571 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1509 to 0x280000400:1537) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1508 to 0x300000400:1537) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1511 to 0x2c0000400:1537) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1510 to 0x240000400:1537) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802b3975d28: operation mds_close to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: 15657:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 15657:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 383 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1634 to 0x2c0000400:1665) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1616 to 0x280000400:1697) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1625 to 0x300000400:1665) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1626 to 0x240000400:1665) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1763 to 0x300000400:1793) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1763 to 0x2c0000400:1793) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1796 to 0x280000400:1825) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1763 to 0x240000400:1793) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 18812:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete Lustre: 23337:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 23337:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1100 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1877 to 0x240000400:1921) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1879 to 0x2c0000400:1921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1916 to 0x280000400:1953) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1863 to 0x300000400:1889) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802aaf54138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2028 to 0x280000400:2049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:1996 to 0x240000400:2017) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1965 to 0x300000400:1985) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1997 to 0x2c0000400:2017) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: 11815:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88031cbd9940 x1830537589105664/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0 projid:4294967295 LustreError: 11815:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:1988 to 0x300000400:2017) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2027 to 0x2c0000400:2049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2031 to 0x240000400:2049) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2061 to 0x280000400:2081) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 29414:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ef90ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ab074f00/0x699aaef116ca51a3 lrc: 3/0,0 mode: PR/PR res: [0x20000afe1:0x238:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x699aaef116ca5195 expref: 3 pid: 29414 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete Lustre: 336:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 336:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1214 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2066 to 0x240000400:2081) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2098 to 0x280000400:2113) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2066 to 0x2c0000400:2081) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2033 to 0x300000400:2049) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2091 to 0x240000400:2113) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2058 to 0x300000400:2081) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2091 to 0x2c0000400:2113) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2123 to 0x280000400:2145) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.118@tcp (at 0@lo) Lustre: Skipped 67 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2135 to 0x2c0000400:2177) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2103 to 0x300000400:2145) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2167 to 0x280000400:2209) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2136 to 0x240000400:2177) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802aaf54138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 9 previous similar messages LustreError: 8157:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup LustreError: 8157:0:(obd_class.h:479:obd_check_dev()) Skipped 131 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2179 to 0x300000400:2209) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2217 to 0x2c0000400:2241) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2214 to 0x240000400:2241) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2259 to 0x280000400:2305) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 13 times Lustre: Failing over lustre-MDT0000 LustreError: 11813:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88029a8a7840 x1830537594013824/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 LustreError: 11813:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2388 to 0x280000400:2433) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2291 to 0x300000400:2337) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2324 to 0x2c0000400:2369) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2324 to 0x240000400:2369) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 14 times Lustre: Failing over lustre-MDT0000 LustreError: 12844:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880297dc2e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802da412200/0x699aaef116d074ba lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x3a1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x699aaef116d074a5 expref: 4 pid: 12844 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 11818:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1745738763/real 1745738763] req@ffff8802d0455a40 x1830537592871936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1745738848 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 11818:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 122 previous similar messages Lustre: 17674:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 Lustre: 17674:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1767 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2504 to 0x2c0000400:2529) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2453 to 0x240000400:2497) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2415 to 0x300000400:2433) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2553 to 0x280000400:2593) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 15 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2525 to 0x300000400:2561) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2685 to 0x280000400:2721) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2622 to 0x2c0000400:2657) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2589 to 0x240000400:2625) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 16 times Lustre: Failing over lustre-MDT0000 LustreError: 21522:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880297970958 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a62b8b80/0x699aaef116d2ffac lrc: 3/0,0 mode: CW/CW res: [0x20000afe1:0x37e:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x699aaef116d2ff9e expref: 4 pid: 21522 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 11 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:2723 to 0x240000400:2753) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2746 to 0x2c0000400:2785) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:2638 to 0x300000400:2657) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2803 to 0x280000400:2849) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 17 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 19203:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 9603:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880095bf0800 ns: mdt-lustre-MDT0001_UUID lock: ffff880097cf3400/0x16068ade32ee477f lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x47d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.16@tcp remote: 0xbf1db77710c1dcb2 expref: 3 pid: 9603 timeout: 0 lvb_type: 0 LustreError: 19203:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 19203 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c9b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa087b5a9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0855b77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0855cdf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056288b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0855cb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0855cb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0565a46>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0854310>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0854da1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085537b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f0deac>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa05747e2>] obd_precleanup+0x142/0x220 [obdclass] [<ffffffffa055f3c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa057a6a5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa057b207>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa057d40c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa057d4f0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0903764>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa120853b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13546 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.16@tcp (stopping) LustreError: 17060:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup LustreError: 17060:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 8363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 8363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 151 previous similar messages Lustre: Skipped 37 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 33 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 8895:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 8627:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800989a0958 ns: mdt-lustre-MDT0000_UUID lock: ffff88032686d680/0x3b0a85bdbad6a78 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xef8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x3b0a85bdbad6a6a expref: 3 pid: 8627 timeout: 0 lvb_type: 0 LustreError: 8895:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG CPU: 15 PID: 8895 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0184b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a1ea9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff817e2c6b>] ? native_safe_halt+0xb/0x20 [<ffffffffa067eb77>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067ecdf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa038a88b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa067ecb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa067ecb0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa038da46>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa067d310>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa067dda1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa067e37b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa1291a7c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa039ca22>] obd_precleanup+0x142/0x220 [obdclass] [<ffffffffa03873c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03a28e5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03a3447>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a564c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03a5730>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa072c794>] server_put_super+0xb84/0x1220 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15add4b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=16515 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802d88a8008: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 68 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.108@tcp (at 0@lo) Lustre: Skipped 73 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3946 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 19763:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 19763:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 20085:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup LustreError: 20085:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: 9488:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1744791097/real 1744791097] req@ffff8802b4129940 x1829543121716608/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1744791113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 9488:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3952 to 0x300000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3983 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4016 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3988 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3988 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4052 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802d88a8008: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4113 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4049 to 0x300000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4049 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4081 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 25967:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 25967:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4148 to 0x240000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4084 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4116 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4085 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4144 to 0x280000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4208 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4175 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4144 to 0x300000400:4161) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4213 to 0x2c0000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4181 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4180 to 0x280000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4245 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4305 to 0x240000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4241 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4274 to 0x2c0000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4242 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 9485:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802e0886940 x1829543121017088/t317827580346(317827580346) o101->lustre-MDT0000-mdc-ffff8802d88a8008@0@lo:12/10 lens 576/608 e 0 to 0 dl 1744791315 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 9485:0:(client.c:3262:ptlrpc_replay_interpret()) Skipped 272 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4305 to 0x2c0000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4274 to 0x300000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4273 to 0x280000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4338 to 0x240000400:4353) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4369 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4306 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4337 to 0x2c0000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4305 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 15232:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 12716:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009c9aa800 ns: mdt-lustre-MDT0000_UUID lock: ffff88008ecee000/0x7968abb5ba372b16 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x10ef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.39@tcp remote: 0x3f4021011032e692 expref: 3 pid: 12716 timeout: 0 lvb_type: 0 LustreError: 15232:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 15232 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0272b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0801d09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa07dc787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa07dc8ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa048834b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa07dc8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa07dc8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa048b506>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa07daf20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa07db9b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa07dbf8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa136ca0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa049a2a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0484e69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa04a0195>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04a0cf7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa04a2efc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa04a2fe0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0888694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166653b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5912 ... LustreError: 4868:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 4868:0:(osd_handler.c:698:osd_ro()) Skipped 13 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) LustreError: 5514:0:(obd_class.h:473:obd_check_dev()) Device 13 not setup LustreError: 5514:0:(obd_class.h:473:obd_check_dev()) Skipped 107 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 16 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 19 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5505 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5452 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: 3290:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1744727863/real 1744727863] req@ffff88012b70c700 x1829476690968832/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1744727879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3290:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 70 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5546 to 0x280000400:5569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5610 to 0x240000400:5633) Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 10107:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880136a32000 ns: mdt-lustre-MDT0000_UUID lock: ffff880130375200/0x7968abb5ba36372e lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xedc:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.39@tcp remote: 0x3f4021011032a08b expref: 3 pid: 10107 timeout: 0 lvb_type: 0 LustreError: 16103:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880090e14a80 x1829476691264768/t0(0) o105->lustre-MDT0000@192.168.203.39@tcp:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 16103:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5645 to 0x280000400:5665) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5708 to 0x240000400:5729) Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 25764:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7208:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8801302d8000 ns: mdt-lustre-MDT0000_UUID lock: ffff880071c06000/0x45e3f881e5c9ced0 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x104c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.45@tcp remote: 0xb2d08161722c3d59 expref: 3 pid: 7208 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) LustreError: 25764:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 25764 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02ceb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0810d09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa07eb787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa07eb8ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa049734b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa07eb8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa07eb8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa049a506>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa07e9f20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa07ea9b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa07eaf8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f29a0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa04a92a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0493e69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa04af195>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04afcf7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa04b1efc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa04b1fe0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0897694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa122353b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13521 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) LustreError: 17227:0:(obd_class.h:473:obd_check_dev()) Device 22 not setup LustreError: 17227:0:(obd_class.h:473:obd_check_dev()) Skipped 75 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 23140:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 23140:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 144 previous similar messages Lustre: Skipped 42 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5094 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5057 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20175:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5440 to 0x280000401:5473) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5407 to 0x2c0000401:5441) Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24447:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: 7208:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff880097d34e00 x1829422335153152/t0(0) o104->lustre-MDT0000@192.168.203.45@tcp:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 8980:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 8980:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 8980 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01efb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0876d09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0851787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa08518ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa055e34b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa08518c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa08518c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0561506>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa084ff20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08509b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0850f8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa1367a0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa05702a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa055ae69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0576195>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0576cf7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa0578efc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0578fe0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08fd694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1660dab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=6778 ... LustreError: 8196:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 8196:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 30059:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a569e000 ns: mdt-lustre-MDT0000_UUID lock: ffff88009bcef400/0x604d1718e1188395 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.20@tcp remote: 0x75ed43ac51b62357 expref: 4 pid: 30059 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18716:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 16713:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88008be08000 ns: mdt-lustre-MDT0000_UUID lock: ffff88012b502000/0x5224afce1f593015 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x118f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.33@tcp remote: 0x58c68522fb04ff40 expref: 3 pid: 16713 timeout: 0 lvb_type: 0 LustreError: 18716:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 18716 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0279b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa086fd09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa084a787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa084a8ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa055834b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa084a8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa084a8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa055b506>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0848f20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08499b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0849f8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa1366a0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa056a2a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0554e69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0570195>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0570cf7>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa0572efc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0572fe0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f6694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166053b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5736 ... LustreError: 4662:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 4662:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) Lustre: Skipped 1 previous similar message LustreError: 11044:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880135993480 x1829121460675200/t0(0) o105->lustre-MDT0000@192.168.202.33@tcp:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 26840:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012ceac000 ns: mdt-lustre-MDT0000_UUID lock: ffff880095393000/0x5224afce1f554c4e lrc: 5/0,0 mode: PR/PR res: [0x20001a9e3:0xf40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.33@tcp remote: 0x58c68522fb03dbe7 expref: 3 pid: 26840 timeout: 0 lvb_type: 0 LustreError: 5394:0:(obd_class.h:473:obd_check_dev()) Device 16 not setup LustreError: 5394:0:(obd_class.h:473:obd_check_dev()) Skipped 35 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5456 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5510 to 0x240000400:5537) Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: 3295:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1744388958/real 1744388958] req@ffff88012cbe1f80 x1829121460675584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1744388974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3295:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 76 previous similar messages Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5624 to 0x240000400:5665) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5560 to 0x280000400:5601) Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5735 to 0x240000400:5761) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5671 to 0x280000400:5697) Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5833 to 0x240000400:5857) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5769 to 0x280000400:5793) Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 27148:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 24550:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012c00b800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800b4de5800/0x7f51405589096491 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xec2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.60@tcp remote: 0x1a0b64856d4df4c expref: 3 pid: 24550 timeout: 0 lvb_type: 0 LustreError: 27148:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 27148 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa028fb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0878159>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0851317>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa085147f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa055e6ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0851450>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0851450>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0561866>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa084fab0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0850541>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0850b1b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa1363f2c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05704c2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa055b0c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0576415>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0577067>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05792ac>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0579390>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08fbeb4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1666c2b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5964 ... LustreError: 6594:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 6594:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 33 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: 3309:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1744150690/real 1744150690] req@ffff8800a5b02d80 x1828871436157568/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1744150706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3309:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 81 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 18 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 18 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5506 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5452 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11294:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 11294:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5544 to 0x280000400:5569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5608 to 0x240000400:5633) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5645 to 0x280000400:5665) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5709 to 0x240000400:5729) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5889) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5739 to 0x280000400:5793) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5955 to 0x240000400:5985) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5858 to 0x280000400:5889) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6049 to 0x240000400:6081) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5954 to 0x280000400:5985) Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 32625:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 32625:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 11 PID: 32625 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01a9b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d2d09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06ad787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06ad8ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03bb38b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06ad8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06ad8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03be546>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06abf20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06ac9b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06acf8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f90a0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03cd2e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03b7ea9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03d31d5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03d3d37>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d5f3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03d6020>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0759694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa17a94db>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=30222 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031e73c138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3919 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88031e73c138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 26 previous similar messages LustreError: 30589:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f1d02548 ns: mdt-lustre-MDT0000_UUID lock: ffff8802c92ba200/0x4c4dc1a22dce2712 lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xef7:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x4c4dc1a22dce2704 expref: 3 pid: 30589 timeout: 0 lvb_type: 0 LustreError: 22889:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8803238a0f40 x1828616595030656/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 22889:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 30962:0:(obd_class.h:473:obd_check_dev()) Device 24 not setup LustreError: 30962:0:(obd_class.h:473:obd_check_dev()) Skipped 55 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.93@tcp (at 0@lo) Lustre: Skipped 24 previous similar messages Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4028 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4028 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4093 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4155 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4091 to 0x300000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4092 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4091 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-91.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 32353:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031e73ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e0ebb4c0/0x4c4dc1a22dd0d24f lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf0f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x4c4dc1a22dd0d241 expref: 3 pid: 32353 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 9175:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 6411:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802dea54138 ns: mdt-lustre-MDT0000_UUID lock: ffff88009ad007c0/0xfc7a8143d1a4ce63 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1190:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xfc7a8143d1a4ce55 expref: 3 pid: 6411 timeout: 0 lvb_type: 0 LustreError: 9175:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 9175 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0185b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06dcd09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06b7787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06b78ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03c538b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06b78c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06b78c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03c8546>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06b5f20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06b69b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06b6f8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12b0a0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa03d72e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03c1ea9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03dd1d5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ddd37>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03dff3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03e0020>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0763694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15cc4db>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=14211 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802f76dca88: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 63 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.73@tcp (at 0@lo) Lustre: Skipped 68 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3913 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.73@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3957 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3957 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3956 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4020 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 20930:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 20930:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f76dca88: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 21310:0:(obd_class.h:473:obd_check_dev()) Device 21 not setup LustreError: 21310:0:(obd_class.h:473:obd_check_dev()) Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4018 to 0x280000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4019 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4019 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4083 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4085 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4150 to 0x240000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4054 to 0x280000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4085 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4208 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4144 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4144 to 0x2c0000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4112 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 4794:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743817456/real 1743817456] req@ffff8800a5bfc140 x1828522156399360/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743817472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 4794:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 66 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4153 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4185 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4186 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4249 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4207 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4239 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4239 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4303 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4346 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4276 to 0x2c0000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4250 to 0x280000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4270 to 0x300000400:4289) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 4780:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802b5e61940 x1828522154141312/t317827580344(317827580344) o101->lustre-MDT0000-mdc-ffff8802f76dca88@0@lo:12/10 lens 576/608 e 0 to 0 dl 1743817587 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 4780:0:(client.c:3262:ptlrpc_replay_interpret()) Skipped 302 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4333 to 0x2c0000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4398 to 0x240000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4302 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4301 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 3mdts recovery; 1 clients	LustreError: 20532:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 20532:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 20532 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa017eb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a4d09>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa067f787>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa067f8ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa038d38b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa067f8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa067f8c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0390546>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa067df20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa067e9b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa067ef8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa127da0c>] mdt_device_fini+0xdc/0x8f0 [mdt] [<ffffffffa039f2e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0389ea9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03a51d5>] class_cleanup+0x245/0x860 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03a5d37>] class_process_config+0x547/0x2660 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a7f3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03a8020>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa072b694>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15994db>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=8588 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 21099:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 21099:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 132 previous similar messages Lustre: Skipped 64 previous similar messages LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.63@tcp (at 0@lo) Lustre: Skipped 55 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3745) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3748 to 0x2c0000402:3777) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3745) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3745) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) lustre-MDT0001-mdc-ffff8800aca65d28: can't stat MDS #0: rc = -107 LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) lustre-MDT0001-mdc-ffff8800aca65d28: can't stat MDS #0: rc = -19 LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) lustre-MDT0001-mdc-ffff8800aca65d28: can't stat MDS #0: rc = -19 LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) lustre-MDT0001-mdc-ffff8800aca65d28: can't stat MDS #0: rc = -19 Lustre: 17837:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.123.63@tcp: no remote llog for lustre-sptlrpc, check MGS config LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) lustre-MDT0001-mdc-ffff8800aca65d28: can't stat MDS #0: rc = -11 LustreError: 8893:0:(lmv_obd.c:1442:lmv_statfs()) Skipped 2 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:225) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:225) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:225) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:225) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 LustreError: 3939:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88029f3e92a8 ns: mdt-lustre-MDT0002_UUID lock: ffff8802cb3f8040/0x404c068b6ac34c58 lrc: 4/0,0 mode: CW/CW res: [0x2800007ed:0x1cf:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x404c068b6ac34c4a expref: 4 pid: 3939 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18125:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 15589:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880091306000 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a5c34400/0xabd2c65db05a5264 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xeeb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.24@tcp remote: 0xd3df5ee023e05bb8 expref: 3 pid: 15589 timeout: 0 lvb_type: 0 LustreError: 18125:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 18125 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01e2b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0882159>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa085b317>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa085b47f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa05686ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa085b450>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa085b450>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa056b866>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0859ab0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa085a541>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085ab1b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa136af2c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057a4c2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa05650c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0580415>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0581067>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05832ac>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0583390>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0905eb4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166dc2b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5709 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 20 previous similar messages Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5456 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5510 to 0x240000400:5537) Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743436649/real 1743436649] req@ffff8800a046d880 x1828122955370112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743436665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3294:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 64 previous similar messages Lustre: Failing over lustre-MDT0000 LustreError: 3296:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880099b6aa00 x1828122955517696/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5619 to 0x240000400:5665) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5556 to 0x280000400:5601) Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 11000:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 11000:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5675 to 0x280000400:5697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5738 to 0x240000400:5761) Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5768 to 0x280000400:5793) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5831 to 0x240000400:5857) Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 3327:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1549:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031f8453d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ac4b8f40/0x6a1b5eeedd9a47dd lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x6a1b5eeedd9a47cf expref: 3 pid: 1549 timeout: 0 lvb_type: 0 LustreError: 3327:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 3327 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018cb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06941c9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa066d387>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa066d4ef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa037a6ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa066d4c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa066d4c0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa037d866>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa066bb20>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa066c5b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa066cb8b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12bdf2c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa038c452>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03770c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03923a5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0392ff7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa039523c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0395320>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0717f24>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15c0beb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13879 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800a38eefc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.3@tcp (at 0@lo) Lustre: Skipped 63 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3918 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3917 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3918 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3960 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3970 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3958 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4011 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 20462:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 20462:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800a38eefc8: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4013 to 0x2c0000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4044 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4013 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4013 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4087 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4053 to 0x280000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4060 to 0x300000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4052 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4147 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4118 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4111 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4115 to 0x280000400:4161) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 27688:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ea640958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802a6d50400/0x6a1b5eeedd98dc23 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x109d:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x6a1b5eeedd98dc15 expref: 4 pid: 27688 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.3@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4185 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4185 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4153 to 0x2c0000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4217 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 3918:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743281539/real 1743281539] req@ffff8802de807840 x1827960185966464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743281555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3918:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 92 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4278 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4246 to 0x300000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4263 to 0x280000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4221 to 0x2c0000400:4257) Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 30424:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 27791:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009bfe5000 ns: mdt-lustre-MDT0000_UUID lock: ffff880094f1be00/0xed5b643e8b7fcd3b lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xee4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.23@tcp remote: 0x67f0390c64643b70 expref: 3 pid: 27791 timeout: 0 lvb_type: 0 LustreError: 30424:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 30424 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01d1b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0884489>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa085d657>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa085d79f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056c61b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa085d770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa085d770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa056f7d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa085c250>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa085cce1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085d2ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa13731dc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057e492>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0569039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05843e5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0585037>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058727c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0587360>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0908054>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1671d7b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=6291 ... LustreError: 7000:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 7000:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5508 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5455 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 3302:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1742983716/real 1742983716] req@ffff8801364cea00 x1827647427168768/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1742983732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3302:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5613 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5549 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5647 to 0x280000400:5665) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5711 to 0x240000400:5729) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5794 to 0x240000400:5825) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5730 to 0x280000400:5761) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 21431:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 21431:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5891 to 0x240000400:5921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5827 to 0x280000400:5857) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5983 to 0x240000400:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5919 to 0x280000400:5953) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6018 to 0x280000400:6049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6082 to 0x240000400:6113) Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 4281:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 4281:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 4281 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa015fb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06ba2e9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0693447>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06935af>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03a06ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0693580>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0693580>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a3866>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0691be0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0692671>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0692c4b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0deaf2c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03b24c2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa039d0c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b8405>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03b9057>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03bb29c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03bb380>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa073e0c4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa10efbfb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=679 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802efeeefc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 31387:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f1df8008 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f9517c00/0x5497d53e2e9548db lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf29:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x5497d53e2e9548cd expref: 4 pid: 31387 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0002: Connection restored to 192.168.123.73@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4030 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3966 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3965 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3965 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802efeeefc8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4093 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4030 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4029 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 3576:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802e9ceb240 x1827634994174592/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 LustreError: 3576:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 9 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4091 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4092 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4156 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4154 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4218 to 0x240000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4154 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4153 to 0x300000400:4193) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.73@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4282 to 0x240000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4219 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4218 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4218 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 4010:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ad52b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff880320c0f480/0x5497d53e2e9ab54a lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x119b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5497d53e2e9ab53c expref: 4 pid: 4010 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 840:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 451:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880326c29bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a29125c0/0x2210a2fd8a9d8ede lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2210a2fd8a9d8ed0 expref: 3 pid: 451 timeout: 0 lvb_type: 0 LustreError: 840:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 840 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01adb4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06bb2e9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0694447>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06945af>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03a16ab>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0694580>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0694580>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a4866>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0692be0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0693671>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0693c4b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f03f2c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03b34c2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa039e0c9>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b9405>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ba057>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03bc29c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03bc380>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa073f0c4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1787bfb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=28093 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 28834:0:(ldlm_resource.c:982:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e3:0xef6:0x0].0x17a4f95f (ffff8802f53cca40) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3966 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 29017:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031de06678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802c9038b80/0x2210a2fd8a969c28 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x101b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2210a2fd8a969c1a expref: 2 pid: 29017 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-mdc-ffff8802eac41bf8: Connection restored to 192.168.123.33@tcp (at 0@lo) Lustre: Skipped 33 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4031 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4095 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4030 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4031 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 7669:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1742773536 with bad export cookie 2454641006752472363 LustreError: 29681:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031be84138 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f7fe8040/0x2210a2fd8a97aee8 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x1097:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2210a2fd8a97aeda expref: 3 pid: 29681 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: lustre-MDT0000-mdc-ffff8802eac41bf8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 37 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4097 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4097 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4096 to 0x280000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4161 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 30132:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b3e3c138 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e260c3c0/0x2210a2fd8a98df30 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xef9:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2210a2fd8a98df22 expref: 4 pid: 30132 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4227 to 0x240000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4163 to 0x280000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4164 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4163 to 0x300000400:4193) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 5233:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009de49440 x1827427719168256/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.33@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4288 to 0x240000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4224 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4224 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4223 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 5246:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d8551940 x1827427719906560/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 LustreError: 5246:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: 5236:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1742773609/real 1742773609] req@ffff88031cfcc640 x1827427719175040/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1742773625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 5236:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 60 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4284 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4284 to 0x2c0000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4348 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4285 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4347 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4348 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4412 to 0x240000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4347 to 0x2c0000400:4385) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 5230:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88031cfcf340 x1827427714486528/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8802eac41bf8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1742773715 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 5230:0:(client.c:3262:ptlrpc_replay_interpret()) Skipped 481 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4409 to 0x2c0000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4409 to 0x280000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4473 to 0x240000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4409 to 0x300000400:4449) Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 10079:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7776:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880095261800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800950a5c00/0xea128cbb33ca4207 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1381:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.15@tcp remote: 0xc12f1d0efb64714c expref: 4 pid: 7776 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) LustreError: 10079:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 10079 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02af13d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa08827b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30 [<ffffffffa085b9b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa085bb1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056a61b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa085baf0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa085baf0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa056d7d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa085a150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa085abe1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085b1bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa136b29c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057ae52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0567039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0580da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05819f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa0583c3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0583d20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa09066b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166ad2b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=6158 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 29795:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012b5b4000 ns: mdt-lustre-MDT0000_UUID lock: ffff88012cb4da00/0xea128cbb33c2d342 lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xee0:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.15@tcp remote: 0xc12f1d0efb6240ce expref: 3 pid: 29795 timeout: 0 lvb_type: 0 LustreError: 8891:0:(ldlm_resource.c:982:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e3:0xee0:0x0].0x0 (ffff880130b17500) refcount nonzero (2) after lock cleanup; forcing cleanup. LustreError: 7134:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a5bfed80 x1826744084974208/t0(0) o105->lustre-MDT0000@192.168.203.15@tcp:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5438 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5491 to 0x240000400:5537) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5596 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5531 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5684 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5620 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5752 to 0x240000400:5793) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5688 to 0x280000400:5729) Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: 3299:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1742122242/real 1742122242] req@ffff880094fadf80 x1826744085297024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1742122258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3299:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5839 to 0x240000400:5857) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5775 to 0x280000400:5793) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 21110:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009222f800 ns: mdt-lustre-MDT0000_UUID lock: ffff88012cbac600/0xea128cbb33c5a02f lrc: 3/0,0 mode: --/CW res: [0x20001a9e3:0xecd:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.15@tcp remote: 0xc12f1d0efb6311bd expref: 4 pid: 21110 timeout: 0 lvb_type: 0 LustreError: 23232:0:(ldlm_resource.c:982:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e3:0xecd:0x0].0x0 (ffff8800b4843e00) refcount nonzero (0) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5832 to 0x280000400:5857) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5896 to 0x240000400:5921) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 25916:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 25916:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5976 to 0x240000400:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5912 to 0x280000400:5953) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6007 to 0x280000400:6049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6070 to 0x240000400:6113) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6118 to 0x280000400:6145) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6182 to 0x240000400:6209) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6207 to 0x280000400:6241) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6271 to 0x240000400:6305) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 LustreError: 3298:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a475a300 x1826744086211840/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 LustreError: 4404:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009f67d800 ns: mdt-lustre-MDT0000_UUID lock: ffff880130031000/0xea128cbb33c99de2 lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xef5:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.15@tcp remote: 0xc12f1d0efb6441fe expref: 3 pid: 4404 timeout: 0 lvb_type: 0 LustreError: 7157:0:(ldlm_resource.c:982:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e3:0xef5:0x0].0x0 (ffff8800a4a15600) refcount nonzero (0) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6308 to 0x280000400:6337) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6372 to 0x240000400:6401) Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 3881:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 14424:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880094534800 ns: mdt-lustre-MDT0001_UUID lock: ffff88012e335200/0xe9a49e453c80f5f9 lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x34a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.45@tcp remote: 0xdc67833523942321 expref: 3 pid: 14424 timeout: 0 lvb_type: 0 LustreError: 3881:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 3881 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0281b4d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0882999>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa085baf7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa085bc5f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056a60b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa085bc30>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa085bc30>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa056d7c6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa085a290>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa085ad21>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085b2fb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f138fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057ae42>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0567029>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0580d85>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05819d7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa0583c1c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0583d00>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa09066d4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa12150cb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13621 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 14424:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 14424:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 127 previous similar messages LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 38 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20065:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:571 to 0x2c0000400:609) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:588 to 0x280000400:609) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24333:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:997 to 0x280000400:1025) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:996 to 0x2c0000400:1025) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28583:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1431 to 0x280000400:1473) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1431 to 0x2c0000400:1473) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5185) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5153) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 364:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.145@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1861 to 0x280000400:1889) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1860 to 0x2c0000400:1889) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5217) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5185) Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 5982:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 9574:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800808da000 ns: mdt-lustre-MDT0000_UUID lock: ffff88012f0fc600/0xf0a779e74571968a lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xbaf:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.201.39@tcp remote: 0xb99eeaa72592973f expref: 4 pid: 9574 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) Lustre: Skipped 8 previous similar messages LustreError: 5982:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 5982 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c9b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa087d989>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30 [<ffffffffa0856b87>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0856cef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056561b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0856cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0856cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa05687d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0855320>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0855db1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffffa085638b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f2829c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0575e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0562039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa057bda5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa057c9f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa057ec3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa057ed20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa09018d4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1227c1b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13497 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 17794:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 17794:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 130 previous similar messages Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5058 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5094 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20017:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:449) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5326 to 0x2c0000401:5345) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5357 to 0x280000401:5377) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24282:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:481) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5587 to 0x280000401:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5554 to 0x2c0000401:5569) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28539:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:513) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:513) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 7208:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800908b5000 ns: mdt-lustre-MDT0000_UUID lock: ffff88009d95b400/0xf0a779e7456c129e lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x1138:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.201.39@tcp remote: 0xb99eeaa72590e554 expref: 4 pid: 7208 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5800 to 0x2c0000401:5825) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5863 to 0x280000401:5889) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 333:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:545) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:545) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6040 to 0x2c0000401:6081) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6105 to 0x280000401:6145) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4664:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:577) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:577) Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 5287:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3940:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88006fd76000 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a7a62600/0x59c1b7a0319a926d lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x156e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.201.38@tcp remote: 0x7693be88156f40b0 expref: 3 pid: 3940 timeout: 0 lvb_type: 0 LustreError: 5287:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 5287 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02d3b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa087e989>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0857b87>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0857cef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056661b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0857cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0857cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa05697d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0856320>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0856db1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085738b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f0f29c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0576e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0563039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa057cda5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa057d9f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa057fc3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa057fd20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa09028d4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa120ec4b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=10621 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5508 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5454 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) LustreError: 3497:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006dec4700 x1826382217285248/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5557 to 0x280000400:5601) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5620 to 0x240000400:5665) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5743 to 0x240000400:5761) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5679 to 0x280000400:5697) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: 3494:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741777030/real 1741777030] req@ffff88012d614a80 x1826382217639552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741777046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3494:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 14 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5766 to 0x280000400:5793) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5830 to 0x240000400:5857) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5922 to 0x240000400:5953) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5857 to 0x280000400:5889) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5956 to 0x280000400:5985) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6020 to 0x240000400:6049) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6113 to 0x240000400:6145) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6049 to 0x280000400:6081) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6230 to 0x240000400:6273) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6166 to 0x280000400:6209) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6267 to 0x280000400:6305) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6331 to 0x240000400:6369) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6394 to 0x280000400:6433) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6458 to 0x240000400:6497) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6558 to 0x240000400:6593) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6494 to 0x280000400:6529) Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 27225:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 24791:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a6d5d3d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802a5d06580/0xcbb98c365bbb3e82 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xef3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xcbb98c365bbb3e74 expref: 3 pid: 24791 timeout: 0 lvb_type: 0 LustreError: 27225:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 27225 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018313d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa068d7b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06669b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0666b1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa037561b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0666af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0666af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03787d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0665150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0665be1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06661bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa129529c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0385e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0372039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa038bda5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa038c9f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa038ec3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa038ed20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07116b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15b6ccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=1659 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88031a088008: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 69 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.43@tcp (at 0@lo) Lustre: Skipped 68 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 5248:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5248:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.43@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4016 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3951 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3951 to 0x280000400:3969) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031a088008: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 9 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3987 to 0x2c0000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3987 to 0x300000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4051 to 0x240000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 9096:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f51ce678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cd2ed680/0xcbb98c365bb7d11a lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xeea:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xcbb98c365bb7d10c expref: 4 pid: 9096 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4052 to 0x2c0000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4053 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4052 to 0x280000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4117 to 0x240000400:4161) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4117 to 0x300000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4116 to 0x280000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4181 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4117 to 0x2c0000400:4161) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 27024:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741624638/real 1741624638] req@ffff8802ba207840 x1826222785398784/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741624654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 27024:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 16017:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e14c92a8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800aa275680/0xcbb98c365bb92f60 lrc: 3/0,0 mode: --/CW res: [0x20001a9e3:0xedd:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xcbb98c365bb92f52 expref: 4 pid: 16017 timeout: 0 lvb_type: 0 LustreError: 14597:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88028c8e5a40 x1826222785960704/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 14597:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 15 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4185 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4185 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4249 to 0x240000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4185 to 0x280000400:4225) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4306 to 0x240000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4241 to 0x2c0000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4242 to 0x300000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4242 to 0x280000400:4257) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4340 to 0x240000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4277 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4276 to 0x2c0000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4276 to 0x300000400:4321) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 27022:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d0ad6940 x1826222787692032/t0(0) o6->lustre-OST0002-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-2-0.0' uid:0 gid:0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 7227:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 6850:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802cc92c138 ns: mdt-lustre-MDT0000_UUID lock: ffff8800ac6b6940/0x185f270bb687a867 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x108c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x185f270bb687a859 expref: 3 pid: 6850 timeout: 0 lvb_type: 0 LustreError: 7227:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 7227 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01d113d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa079a7b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa07739b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0773b1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa048261b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0773af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0773af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa04857d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0772150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0772be1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa07731bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f2e29c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0492e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa047f039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0498da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04999f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa049bc3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa049bd20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa081e6b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1778ccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5372 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3964 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 17400:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741613283 with bad export cookie 1756165310958064503 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3968 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3968 to 0x300000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4031 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4031 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4095 to 0x240000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4031 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f74defc8: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 13161:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 13161:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 13161 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018013d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa07807b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa07599b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0759b1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa046861b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0759af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0759af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa046b7d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0758150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0758be1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa07591bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12b729c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0478e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0465039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa047eda5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa047f9f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa0481c3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0481d20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08046b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15b6ccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=6110 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802e02492a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 55 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 67 previous similar messages Lustre: 27937:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741524237/real 1741524237] req@ffff8803226dad40 x1826117641771648/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741524253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 27937:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3917 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3917 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 9978:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 9978:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3960 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3960 to 0x2c0000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3960 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4024 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802e02492a8: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages LustreError: 11547:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ad399bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802edac34c0/0x6dc4673ebc5df397 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf01:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x6dc4673ebc5df389 expref: 4 pid: 11547 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 14148:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 11428:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e464ca88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e9680040/0x87df9e81243afccb lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x12a3:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x87df9e81243afcbd expref: 4 pid: 11428 timeout: 0 lvb_type: 0 LustreError: 14148:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 14148 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018413d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06c67b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa069f9b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa069fb1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03ae61b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa069faf0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa069faf0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03b17d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa069e150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa069ebe1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa069f1bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa129e29c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03bee52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03ab039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c4da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03c59f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03c7c3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03c7d20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa074a6b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa159dccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=7955 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8800a1f7c138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 51 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.43@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 11855:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 11855:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.43@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4020 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3956 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3956 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3957 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800a1f7c138: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 10 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4020 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4021 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4021 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4084 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4090 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4090 to 0x2c0000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4154 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4090 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4205 to 0x240000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4141 to 0x280000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4141 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4141 to 0x300000400:4161) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4183 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4248 to 0x240000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4183 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4184 to 0x280000400:4225) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4241 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4240 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4241 to 0x300000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4305 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 28637:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741496373/real 1741496373] req@ffff8802b87b3240 x1826088332980096/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741496389 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 28637:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4344 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4280 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4279 to 0x2c0000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4280 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4409 to 0x240000400:4449) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4345 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4345 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4345 to 0x300000400:4385) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 28626:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a67b4140 x1826088329552128/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8800a1f7c138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1741496475 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 28626:0:(client.c:3262:ptlrpc_replay_interpret()) Skipped 324 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4395 to 0x300000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4395 to 0x2c0000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4459 to 0x240000400:4481) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4395 to 0x280000400:4417) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4443 to 0x300000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4442 to 0x2c0000400:4481) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4442 to 0x280000400:4481) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4507 to 0x240000400:4545) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4499 to 0x300000400:4545) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4564 to 0x240000400:4609) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4500 to 0x280000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4500 to 0x2c0000400:4545) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 13 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 15406:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 15031:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ab926fc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f2c84f00/0xaf2a79a23615f8bd lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xaf2a79a23615f8af expref: 3 pid: 15031 timeout: 0 lvb_type: 0 LustreError: 15406:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 15406 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01c113d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06bb7b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06949b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0694b1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03a361b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0694af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0694af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03a67d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0693150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0693be1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06941bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f2629c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03b3e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03a0039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b9da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03ba9f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03bcc3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03bcd20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa073f6b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1770ccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=9115 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3963 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3919 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4063 to 0x240000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 18697:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f87c7340 x1826028240437888/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.53@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4028 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4125 to 0x240000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4029 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802e1cf0958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 18694:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d5c0f840 x1826028241163392/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 LustreError: 18694:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4186 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4090 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4090 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4089 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4162 to 0x2c0000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4162 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4257 to 0x240000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4162 to 0x300000400:4193) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.53@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4218 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4219 to 0x2c0000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4218 to 0x300000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4314 to 0x240000400:4353) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 18688:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741438918/real 1741438918] req@ffff8802f3bda340 x1826028242856064/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741438934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 18688:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 62 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4283 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4282 to 0x280000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4378 to 0x240000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4282 to 0x2c0000400:4321) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802e1cf0958: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4349 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4445 to 0x240000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4349 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4350 to 0x300000400:4385) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 18684:0:(client.c:3262:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800ab72ad40 x1826028238552960/t317827580344(317827580344) o101->lustre-MDT0000-mdc-ffff8802e1cf0958@0@lo:12/10 lens 576/608 e 0 to 0 dl 1741439019 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 18684:0:(client.c:3262:ptlrpc_replay_interpret()) Skipped 455 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4415 to 0x280000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4415 to 0x2c0000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4511 to 0x240000400:4545) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4415 to 0x300000400:4449) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4577 to 0x240000400:4609) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4481 to 0x2c0000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4481 to 0x280000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4481 to 0x300000400:4513) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 LustreError: 14478:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802dd57b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800ab7b07c0/0xaf2a79a23614e8ff lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x1387:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xaf2a79a23614e8f1 expref: 60 pid: 14478 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 22444:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1741439035 with bad export cookie 12622034643258176420 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4638 to 0x240000400:4673) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4542 to 0x300000400:4577) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4542 to 0x2c0000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4543 to 0x280000400:4577) Lustre: DEBUG MARKER: centos-51.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 3820:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 1300:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031f0f6678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b34ed2c0/0x1243eaf8e101f851 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x1243eaf8e101f843 expref: 3 pid: 1300 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 3820:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 15 PID: 3820 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa016a13d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa069a7b9>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa06739b7>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0673b1f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa038261b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0673af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0673af0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03857d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0672150>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0672be1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06731bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa12b829c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0392e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa037f039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0398da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03999f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa039bc3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa039bd20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa071e6b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15d9ccb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=18512 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88031d7cdd28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 65 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.78@tcp (at 0@lo) Lustre: Skipped 67 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3917 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3917 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3917 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 9850:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1741417169/real 1741417169] req@ffff880097046e40 x1826005365460352/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1741417185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 9850:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 138 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4029 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3965 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3965 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3964 to 0x2c0000400:4001) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 26430:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 26430:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031d7cdd28: operation mds_sync to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4017 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4016 to 0x2c0000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4080 to 0x240000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4016 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4057 to 0x280000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4057 to 0x2c0000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4121 to 0x240000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4057 to 0x300000400:4097) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 9847:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800ac29c140 x1826005367821952/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4183 to 0x240000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4119 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4119 to 0x2c0000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4119 to 0x280000400:4161) Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 8802:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7203:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b3e0a000 ns: mdt-lustre-MDT0000_UUID lock: ffff880094cf0400/0x6cc6505f39e72cff lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xee4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.29@tcp remote: 0xcc9fcb7b1b76c2fe expref: 4 pid: 7203 timeout: 0 lvb_type: 0 LustreError: 8802:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 8802 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02b3b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0879489>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30 [<ffffffffa0852657>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa085279f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa056261b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0852770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0852770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa05657d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0851250>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0851ce1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08522ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa13591dc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0572e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa055f039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0578da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05799f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa057bc3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa057bd20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08fd054>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1657bdb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5770 ... LustreError: 5352:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5352:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5503 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5450 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 5693:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 5693:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 13 PID: 5693 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01ebb9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0701989>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06dab87>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa06dacef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa03e961b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa06dacc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa06dacc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03ec7d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa06d9320>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06d9db1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06da38b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa0f15bcc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03f9e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03e6039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03ffda5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04009f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa0402c3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0402d20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07858d4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa17589fb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=3272 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.38@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3921 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3966 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3922 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4032 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3968 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3968 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4030 to 0x300000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4030 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4029 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4093 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f6de6678: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 38 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4093 to 0x2c0000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4093 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4093 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4157 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-36.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 5320:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f2342548 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e03ea5c0/0x15a01bb7a4c90a45 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1107:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x15a01bb7a4c90a37 expref: 3 pid: 5320 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 32383:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 32383:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 4 PID: 32383 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0187b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0689989>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc] [<ffffffffa0662b87>] cleanup_resource+0x1e7/0x320 [ptlrpc] [<ffffffffa0662cef>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa037161b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa0662cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa0662cc0>] ? cleanup_resource+0x320/0x320 [ptlrpc] [<ffffffffa03747d6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0661320>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0661db1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa066238b>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc] [<ffffffffa1287bcc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0381e52>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa036e039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0387da5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03889f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa038ac3c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa038ad20>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa070d8d4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa158a9fb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=12460 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3979 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 28 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4016 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3953 to 0x2c0000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4017 to 0x300000400:4033) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f4401bf8: operation mds_sync to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4051 to 0x300000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4051 to 0x240000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3987 to 0x2c0000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4111 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4047 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4047 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4111 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 20947:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802c5982548 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d7d38040/0xd0082d6fa6e48ace lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xee1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd0082d6fa6e48ac0 expref: 3 pid: 20947 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4147 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4084 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4147 to 0x300000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4084 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4143 to 0x280000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4207 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4143 to 0x2c0000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4206 to 0x300000400:4225) Lustre: 955:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740825942/real 1740825942] req@ffff88009bc22340 x1825385408480512/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1740825958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 955:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 84 previous similar messages Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4179 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4244 to 0x240000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4179 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4243 to 0x300000400:4289) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4238 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4302 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4238 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4303 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 31698:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 31698:0:(osd_handler.c:698:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 LustreError: 30119:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88029e5c5d28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d7d3c000/0xd0082d6fa6e6bfac lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf10:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xd0082d6fa6e6bf9e expref: 3 pid: 30119 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 25552:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25552:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 25552 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0181b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0741489>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa071a657>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa071a79f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0426e1b>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass] [<ffffffffa071a770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa071a770>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0429fd6>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass] [<ffffffffa0719250>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0719ce1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa071a2ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12af40c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0437652>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0423839>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa043d5a5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa043e1f7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa044043c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0440520>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07c9ec4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15b3b8b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=26533 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3915 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3960 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.23@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3951 to 0x300000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4016 to 0x240000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 32191:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 32191:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802aea72e98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802aea72e98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4050 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3987 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3987 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4046 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4078 to 0x240000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4047 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4046 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4115 to 0x240000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4083 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4083 to 0x300000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4084 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 14119:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740599651/real 1740599651] req@ffff8802ca15f840 x1825148182534912/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1740599667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 14119:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 68 previous similar messages LustreError: MGC192.168.123.23@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4171 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4140 to 0x2c0000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4140 to 0x280000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4139 to 0x300000400:4161) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4216 to 0x240000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4183 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4184 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4183 to 0x300000400:4225) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4242 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4242 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4242 to 0x300000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4274 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 14118:0:(client.c:3268:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802bb2e0a40 x1825148180590080/t317827580344(317827580344) o101->lustre-MDT0000-mdc-ffff8802aea72e98@0@lo:12/10 lens 576/608 e 0 to 0 dl 1740599776 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 14118:0:(client.c:3268:ptlrpc_replay_interpret()) Skipped 237 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4311 to 0x240000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4279 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4278 to 0x2c0000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4279 to 0x300000400:4321) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 14123:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88028e362840 x1825148185076992/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4337 to 0x280000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4337 to 0x2c0000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4337 to 0x300000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4369 to 0x240000400:4385) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4401 to 0x240000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4369 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4369 to 0x280000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4369 to 0x2c0000400:4385) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4398 to 0x280000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4399 to 0x300000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4399 to 0x2c0000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4431 to 0x240000400:4449) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 13 times Lustre: Failing over lustre-MDT0000 LustreError: 24446:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880327774138 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a84f34c0/0x5038bce785120028 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xef8:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5038bce78512001a expref: 3 pid: 24446 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18068:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 15808:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880099f1a800 ns: mdt-lustre-MDT0000_UUID lock: ffff88009fd29400/0x87b1dd457e75c76b lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x10f2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.55@tcp remote: 0x792ffff298c3fc17 expref: 3 pid: 15808 timeout: 0 lvb_type: 0 LustreError: 18068:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 18068 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa020eb9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa08b00b9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0889f07>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa088a04f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0214a1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa088a020>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa088a020>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0217b16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08886a0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0889131>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa088970a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa136f82c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05af3f2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa05a0039>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05b5345>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05b5f97>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05b81dc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa05b82c0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0931a44>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166dbfb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5647 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 27644:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012a32f800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a524ce00/0x87b1dd457e72e3cc lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xeed:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.55@tcp remote: 0x792ffff298c3242f expref: 4 pid: 27644 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5493 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5439 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 8307:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 8307:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5532 to 0x280000400:5569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5595 to 0x240000400:5633) Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 10405:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880092405800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800b1c42600/0x87b1dd457e7453ed lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xef0:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.55@tcp remote: 0x792ffff298c390d2 expref: 4 pid: 10405 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5696 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5632 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5781 to 0x240000400:5825) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5718 to 0x280000400:5761) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 3304:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1740255226/real 1740255226] req@ffff8800a5d72a00 x1824786859569024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1740255242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3304:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 5168:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 5168:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 5168 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01eeb9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d29e9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06acb17>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06acc5f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01f4b1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06acc30>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06acc30>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01f7c16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06ab710>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06ac1a1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06ac77a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f0e67c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03cdbf2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03be839>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03d3b45>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03d4797>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d69dc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03d6ac0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0758514>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1750bab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=31278 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3924 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3969 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3925 to 0x280000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3925 to 0x300000400:3969) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 31643:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f8e64138 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e6dad2c0/0xb32453fed2e0288a lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xef7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb32453fed2e0287c expref: 3 pid: 31643 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3997 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4029 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3996 to 0x2c0000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3997 to 0x300000400:4033) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 8441:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800b3c27340 x1824439753163392/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 LustreError: lustre-MDT0000-mdc-ffff8802ea264138: operation mds_close to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802ea264138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4062 to 0x2c0000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4093 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4061 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4061 to 0x280000400:4097) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4127 to 0x2c0000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4127 to 0x300000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4159 to 0x240000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4128 to 0x280000400:4161) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 1021:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d5b312a8 ns: mdt-lustre-MDT0000_UUID lock: ffff88031dee3100/0xb32453fed2e2f7bc lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x1107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb32453fed2e2f7ae expref: 2 pid: 1021 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4191 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4190 to 0x280000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4222 to 0x240000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4190 to 0x2c0000400:4225) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 8455:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1739924035/real 1739924035] req@ffff8802f1eeee40 x1824439754783744/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1739924051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 8455:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 61 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4255 to 0x280000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4256 to 0x2c0000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4255 to 0x300000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4288 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4320 to 0x280000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4321 to 0x2c0000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4353 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4320 to 0x300000400:4353) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4372 to 0x280000400:4417) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4371 to 0x2c0000400:4417) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4371 to 0x300000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4403 to 0x240000400:4449) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 8439:0:(client.c:3268:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880322933240 x1824439751155456/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8802ea264138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1739924161 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 8439:0:(client.c:3268:ptlrpc_replay_interpret()) Skipped 494 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4480 to 0x240000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4449 to 0x280000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4448 to 0x2c0000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4448 to 0x300000400:4481) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4516 to 0x280000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4515 to 0x300000400:4545) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4547 to 0x240000400:4577) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4515 to 0x2c0000400:4545) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4570 to 0x2c0000400:4609) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4571 to 0x300000400:4609) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4571 to 0x280000400:4609) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4603 to 0x240000400:4641) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000 LustreError: 4781:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f38b4a88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ec228400/0xb32453fed2e9efa4 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1415:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xb32453fed2e9ef96 expref: 3 pid: 4781 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18228:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 17854:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e3566678 ns: mdt-lustre-MDT0000_UUID lock: ffff8803272370c0/0x32c9d04a0a8e6e36 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x32c9d04a0a8e6e28 expref: 3 pid: 17854 timeout: 0 lvb_type: 0 LustreError: 18228:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 5 PID: 18228 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01f2b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06cd9e9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06a7b17>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06a7c5f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01f8b1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06a7c30>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06a7c30>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01fbc16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06a6710>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a71a1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a777a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f2967c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03c8bf2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03b9839>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03ceb45>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03cf797>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03d19dc>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03d1ac0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0753514>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1775bab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=16935 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802ef885d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 68 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.113@tcp (at 0@lo) Lustre: Skipped 78 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3924 to 0x300000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3924 to 0x2c0000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4032 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3988 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3999 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4062 to 0x280000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4094 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3998 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 25518:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25518:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 25518 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b LustreError: 16861:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009bd1a000 ns: mdt-lustre-MDT0000_UUID lock: ffff88011e614200/0xa03f75bb84d8c130 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xbb4:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.55@tcp remote: 0x891c602519f11245 expref: 4 pid: 16861 timeout: 0 lvb_type: 0 [<ffffffffa02d7a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08700d9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30 [<ffffffffa0849f27>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084a06f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02dda2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084a040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084a040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02e0b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08486c0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0849151>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084972a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f0b82c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa056e122>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa055ed69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0574075>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0574cc7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa0576f0c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0576ff0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f2be4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1209d8b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13434 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 16861:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 16861:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 121 previous similar messages Lustre: Skipped 24 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.155@tcp (at 0@lo) Lustre: Skipped 33 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5084 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5120 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19928:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.155@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5494 to 0x280000401:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5463 to 0x2c0000401:5505) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24201:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.155@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping)	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 25575:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 21772:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012a3d6800 ns: mdt-lustre-MDT0000_UUID lock: ffff88013048da00/0xca087a159cb16963 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xbb4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.29@tcp remote: 0x3f5027103636fe24 expref: 3 pid: 21772 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) LustreError: 25575:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 25575 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02dca9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08770d9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0850f27>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa085106f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02e2a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0851040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0851040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02e5b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa084f6c0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0850151>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085072a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f1282c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0575122>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0565d69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa057b075>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa057bcc7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa057df0c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa057dff0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f9be4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa1210bdb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=13478 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 2042:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 2042:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 142 previous similar messages Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.204.129@tcp (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5067 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5103 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19990:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5401 to 0x2c0000401:5441) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5433 to 0x280000401:5473) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24258:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.129@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 20354:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 19983:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d846ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f93734c0/0x3c625924bce41822 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x128b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x3c625924bce41814 expref: 2 pid: 19983 timeout: 0 lvb_type: 0 LustreError: 20354:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 4 PID: 20354 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0211b9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06c90d9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa06a2f27>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06a306f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0217a1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06a3040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06a3040>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa021ab16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06a16c0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a2151>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a272a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0ee782c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03c7122>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03b7d69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03cd075>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03cdcc7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03cff0c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03cfff0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa074bbe4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1763d3b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=16279 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff880325276fc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.68@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4033 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3969 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3969 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4034 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4098 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4033 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4034 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff880325276fc8: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 17757:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f4da6678 ns: mdt-lustre-MDT0000_UUID lock: ffff880323425680/0x3c625924bce02ac2 lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0xf07:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x54b07400000000 nid: 0@lo remote: 0x3c625924bce02ab4 expref: 3 pid: 17757 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4096 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4097 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4160 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4097 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4162 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4161 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4161 to 0x2c0000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4226 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 18868:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ef8c5d28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f8b416c0/0x3c625924bce2277d lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x3c625924bce2276f expref: 3 pid: 18868 timeout: 0 lvb_type: 0 LustreError: 25882:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d6c40f40 x1823384548633856/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 LustreError: 25882:0:(client.c:1282:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.68@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4224 to 0x280000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4223 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4288 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4223 to 0x300000400:4257) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4287 to 0x280000400:4321) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4351 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4287 to 0x2c0000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4287 to 0x300000400:4321) Lustre: DEBUG MARKER: centos-66.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 1815:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 26252:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802fb79e678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802a3d07c00/0xd0ca920e382e9462 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf20:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xd0ca920e382e9454 expref: 4 pid: 26252 timeout: 0 lvb_type: 0 LustreError: 1815:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 1815 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0193a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa067ff39>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0659d77>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffff81402ab9>] ? snprintf+0x49/0x70 [<ffffffffa0659ebf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0199a1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0659e90>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0659e90>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa019cb16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0658510>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0658fa1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa065957a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12a449c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa037f0b2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa036fd19>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0385005>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0385c57>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa0387e9c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0387f80>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa07029b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15c30ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=32375 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 27688:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 512:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009b0f2800 ns: mdt-lustre-MDT0001_UUID lock: ffff880088cad200/0x8f4c8f256892d9d0 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0xad8:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.32@tcp remote: 0x3ee37e6daed0829f expref: 4 pid: 512 timeout: 0 lvb_type: 0 LustreError: 27688:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 27688 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0220a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0871139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa084af87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084b0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0226a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084b0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084b0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0229b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0849720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084a1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084a78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f0bc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05835a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056b549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05894f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058a147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058c38c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa058c470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f3bf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa12099eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 192.168.203.32@tcp (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13545 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 512:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 512:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 140 previous similar messages LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.203.132@tcp (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19976:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.132@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (19976) used greatest stack depth: 9792 bytes left Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:565 to 0x2c0000400:609) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:582 to 0x280000400:609) Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.32@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24236:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.132@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:986 to 0x280000400:1025) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:986 to 0x2c0000400:1025) Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 2596:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2223:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f2638958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e003a5c0/0xd694fb7dcf196d93 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e2:0x10a7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xd694fb7dcf196d85 expref: 4 pid: 2223 timeout: 0 lvb_type: 0 LustreError: 2596:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 2596 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0165a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a5139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa067ef87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa067f0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa016ba1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa067f0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa067f0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa016eb16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa067d720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa067e1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa067e78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0e0282c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03a3232>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0393e79>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03a9185>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03a9dd7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03ac01c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03ac100>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0727c44>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa1100d3b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=703 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802f21c8008: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 50 previous similar messages LustreError: 31644:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f2218958 ns: mdt-lustre-MDT0000_UUID lock: ffff880324058040/0xd694fb7dcf167993 lrc: 3/0,0 mode: --/CW res: [0x20001a9e2:0xf3a:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xd694fb7dcf167985 expref: 4 pid: 31644 timeout: 0 lvb_type: 0 LustreError: 7785:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ea6b1940 x1822998169310464/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.98@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3922 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3966 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3922 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3922 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f21c8008: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 1109:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ee19a548 ns: mdt-lustre-MDT0000_UUID lock: ffff8800ae2687c0/0xd694fb7dcf1758ac lrc: 3/0,0 mode: PR/PR res: [0x20001a9e2:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xd694fb7dcf17589e expref: 3 pid: 1109 timeout: 0 lvb_type: 0 LustreError: 1485:0:(ldlm_resource.c:1149:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e2:0xefe:0x0].0x0 (ffff8802f87a9bc0) refcount nonzero (0) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4030 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4033 to 0x280000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4097 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4034 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4033 to 0x300000400:4065) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 4303:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 3929:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b429ae98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802df5d9a80/0x37d317a0b447c638 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf02:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x37d317a0b447c62a expref: 3 pid: 3929 timeout: 0 lvb_type: 0 LustreError: 4303:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 9 PID: 4303 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa016ba9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a8139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0681f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06820cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0171a1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06820a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06820a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0174b16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0680720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06811b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068178a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0defc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03a6122>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0396d69>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03ac075>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03accc7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03aef0c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03aeff0>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa072ac54>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa10ed9bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=698 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 30936:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8803208137e8 ns: mdt-lustre-MDT0000_UUID lock: ffff880326004780/0x37d317a0b44289db lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xf05:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x37d317a0b44289cd expref: 3 pid: 30936 timeout: 0 lvb_type: 0 mdt00_005 (31951) used greatest stack depth: 9824 bytes left Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.43@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3920 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3965 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3921 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3920 to 0x2c0000400:3937) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.43@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4028 to 0x240000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3964 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3965 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3965 to 0x300000400:4001) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff880320f59bf8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 2212:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 2212:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 14 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4089 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4025 to 0x300000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4024 to 0x280000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4025 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: 2212:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f10d37e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e9771e40/0x37d317a0b4450705 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xeea:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x37d317a0b44506f7 expref: 4 pid: 2212 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4090 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4091 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4090 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4154 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4223 to 0x240000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4158 to 0x280000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4158 to 0x2c0000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4158 to 0x300000400:4193) Lustre: 3817:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1738385870/real 1738385870] req@ffff880318f53240 x1822826700630912/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1738385886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3817:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 51 previous similar messages Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4219 to 0x280000400:4257) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4218 to 0x300000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4218 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4283 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 25516:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 25516:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 25516 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0227a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0875139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] LustreError: 7230:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800994cc000 ns: mdt-lustre-MDT0000_UUID lock: ffff88008ea9d600/0xba5d82080e7d4535 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xfe7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.34@tcp remote: 0x98fdaea8e98640fd expref: 3 pid: 7230 timeout: 0 lvb_type: 0 [<ffffffffa084ef87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084f0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa022da2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084f0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084f0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0230b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa084d720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084e1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084e78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f04c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05875a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056f549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa058d4f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058e147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa059038c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0590470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f7bf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa12029eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13356 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 7204:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7204:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 104 previous similar messages Lustre: Skipped 26 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5095 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5058 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19925:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.134@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5375 to 0x280000401:5409) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5344 to 0x2c0000401:5377) Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24198:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.134@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping)	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 27828:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2102:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009bc4d800 ns: mdt-lustre-MDT0001_UUID lock: ffff880130a72200/0xc7d0c25fcfc2ed36 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0x373:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.23@tcp remote: 0x3d617c062ea2806e expref: 4 pid: 2102 timeout: 0 lvb_type: 0 LustreError: 27828:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 27828 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0221a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08678a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0843f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08440cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02279cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02244c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08440a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08440a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa022aac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0842720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08431b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084378a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f05c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057e382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0566899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05842d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0584f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058729c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058737e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08eb8b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11ff78b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13769 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 10844:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 10844:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 139 previous similar messages Lustre: Skipped 42 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20124:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.123@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (20124) used greatest stack depth: 9792 bytes left Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:586 to 0x280000400:609) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:569 to 0x2c0000400:609) Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24378:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.123@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1039 to 0x2c0000400:1057) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1038 to 0x280000400:1057) Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 25976:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 23612:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800948a1800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800b4c75a00/0x84c868d0489915c9 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xefa:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.13@tcp remote: 0x87c7f3cc83362700 expref: 4 pid: 23612 timeout: 0 lvb_type: 0 LustreError: 25976:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 25976 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c0a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0871139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa084af87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084b0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02c6a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084b0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084b0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02c9b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0849720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084a1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084a78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa1363c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05835a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056b549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05894f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058a147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058c38c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa058c470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f3bf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa16619eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=5773 ... LustreError: 4753:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 4753:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5438 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5491 to 0x240000400:5537) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: 3300:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1738059932/real 1738059932] req@ffff88012e2d9880 x1822484938234880/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1738059948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3300:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 65 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5592 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5528 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: 8979:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a5e86000 ns: mdt-lustre-MDT0000_UUID lock: ffff88009d400200/0x84c868d04895bb46 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xfff:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.13@tcp remote: 0x87c7f3cc83352c65 expref: 4 pid: 8979 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5693 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5629 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5775 to 0x240000400:5793) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5710 to 0x280000400:5729) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5836 to 0x240000400:5857) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5773 to 0x280000400:5793) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5844 to 0x280000400:5889) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5907 to 0x240000400:5953) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6006 to 0x240000400:6049) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5943 to 0x280000400:5985) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 5880:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 5880:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 5880 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b LustreError: 7213:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880078d96800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a838ae00/0x10ba5631fee6d0d0 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xbc1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.9@tcp remote: 0x8da878a1f963061f expref: 4 pid: 7213 timeout: 0 lvb_type: 0 [<ffffffffa0222a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0878139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0851f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08520cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0228a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa08520a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08520a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa022bb26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0850720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08511b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085178a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f12c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa058a5a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0572549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05904f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0591147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa059338c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0593470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08fabf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa12129eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13419 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 15099:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 15099:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 113 previous similar messages Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.109@tcp (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5097 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5060 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19922:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.109@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5365 to 0x280000401:5409) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5332 to 0x2c0000401:5377) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24185:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.109@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5714 to 0x280000401:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5683 to 0x2c0000401:5729) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28447:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.109@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:513) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:513) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5974 to 0x280000401:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5975 to 0x2c0000401:6017) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32705:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.109@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:545) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:545) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6272 to 0x2c0000401:6305) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6272 to 0x280000401:6305) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4562:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.109@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:577) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:577) Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 31622:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 29636:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f08ee678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ccfd07c0/0xad0e3e278deda07f lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xefe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xad0e3e278deda071 expref: 3 pid: 29636 timeout: 0 lvb_type: 0 LustreError: 31622:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 13 PID: 31622 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018ba9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a7139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0680f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06810cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa0191a1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06810a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06810a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0194b16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa067f720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06801b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068078a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12a182c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03a5232>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0395e79>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03ab185>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03abdd7>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03ae01c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03ae100>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0729c44>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa159fd3b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=4192 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802ebddb7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 62 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.118@tcp (at 0@lo) Lustre: Skipped 68 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3916 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3960 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 8727:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 8727:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3966 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3965 to 0x300000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3966 to 0x280000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4029 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 27631:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737823051/real 1737823051] req@ffff8802b4890f40 x1822236653987072/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737823067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 27631:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4012 to 0x300000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4076 to 0x240000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4011 to 0x280000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4012 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4122 to 0x240000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4057 to 0x2c0000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4057 to 0x280000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4057 to 0x300000400:4097) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 16067:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e38f6678 ns: mdt-lustre-MDT0000_UUID lock: ffff880092c20040/0xad0e3e278deab195 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xf0f:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xad0e3e278deab187 expref: 4 pid: 16067 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4173 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4109 to 0x300000400:4129) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4109 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4153 to 0x280000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4154 to 0x300000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4217 to 0x240000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4154 to 0x2c0000400:4193) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4209 to 0x2c0000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4209 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4209 to 0x280000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4273 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802ebddb7e8: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4244 to 0x2c0000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4307 to 0x240000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4244 to 0x300000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4244 to 0x280000400:4289) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4368 to 0x240000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4304 to 0x300000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4304 to 0x280000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4303 to 0x2c0000400:4321) Lustre: DEBUG MARKER: centos-116.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 19002:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 16376:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a5baa000 ns: mdt-lustre-MDT0000_UUID lock: ffff8800ab8b5600/0x9eabe02b4e7538e0 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xee8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.47@tcp remote: 0x8470bf59d5a23223 expref: 3 pid: 16376 timeout: 0 lvb_type: 0 LustreError: 19002:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 19002 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02b0a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086c139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0845f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08460cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02b6a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa08460a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08460a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02b9b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0844720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08451b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084578a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa1357c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057e5a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0566549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05844f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0585147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058738c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0587470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08eebf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa16559eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping)	Lustre: DEBUG MARKER: Started rundbench load pid=5646 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5505 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5452 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 3295:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737731172/real 1737731172] req@ffff8800999f7480 x1822140251280512/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737731188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3295:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 66 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 6352:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a5bb0800 ns: mdt-lustre-MDT0000_UUID lock: ffff88009f7db600/0x9eabe02b4e72569f lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xfcc:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.47@tcp remote: 0x8470bf59d5a15bed expref: 3 pid: 6352 timeout: 0 lvb_type: 0 LustreError: 7054:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880135b50700 x1822140251430016/t0(0) o105->lustre-MDT0000@192.168.203.47@tcp:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 9006:0:(ldlm_resource.c:1149:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x20001a9e3:0xfcc:0x0].0x0 (ffff880092d39d00) refcount nonzero (2) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5612 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 11538:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 11538:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5651 to 0x280000400:5697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5716 to 0x240000400:5761) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5830 to 0x240000400:5857) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5766 to 0x280000400:5793) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 21296:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7210:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880074af0000 ns: mdt-lustre-MDT0000_UUID lock: ffff880092b34800/0xe116af6490cc864c lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xddb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.27@tcp remote: 0xc968a12650cae038 expref: 3 pid: 7210 timeout: 0 lvb_type: 0 LustreError: 21296:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 21296 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02d8a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0870139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0849f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084a0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02dea2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084a0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084a0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02e1b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0848720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08491b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084978a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0effc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05825a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056a549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05884f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0589147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058b38c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa058b470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa08f2d94>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11fd9eb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13528 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 7210:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012d4b9000 ns: mdt-lustre-MDT0000_UUID lock: ffff880072482400/0xe116af6490c95f9a lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xbdf:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.27@tcp remote: 0xc968a12650c9e4d2 expref: 4 pid: 7210 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 15131:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 15131:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 133 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5052 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5088 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 7 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19970:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.127@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (19970) used greatest stack depth: 9776 bytes left Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 5974:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 15103:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009bc02000 ns: mdt-lustre-MDT0000_UUID lock: ffff8800b48c0c00/0xed3e6148bb31fdf0 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x15a6:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.12@tcp remote: 0xfc64905128fadcaf expref: 4 pid: 15103 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) LustreError: 5974:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 5974 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02caa9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0868ec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0845e57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0845f9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02d09cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0845f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0845f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02d3ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08445f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0845081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084565a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f1ee5d>] mdt_device_fini+0xed/0x9b0 [mdt] [<ffffffffa057f8c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0585785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05863d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058874c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058882e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08ec0c1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11e005b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) Lustre: Skipped 4 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13505 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages LustreError: 8365:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 8365:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 111 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5076 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5112 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20009:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.112@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5371 to 0x2c0000401:5409) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5402 to 0x280000401:5441) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24284:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.112@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5659 to 0x2c0000401:5697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5692 to 0x280000401:5729) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28545:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.112@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:513) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:513) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5998 to 0x280000401:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5966 to 0x2c0000401:5985) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 331:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.112@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:545) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:545) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6199 to 0x2c0000401:6241) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6231 to 0x280000401:6273) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4655:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.112@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:577) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:577) Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 1826:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 32256:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b4a75000 ns: mdt-lustre-MDT0000_UUID lock: ffff88012db4da00/0xc841f08ceab0f921 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x13e0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.47@tcp remote: 0x1d270f6ee1f92e7d expref: 3 pid: 32256 timeout: 0 lvb_type: 0 LustreError: 1826:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 1826 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02b1a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086c8a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0848f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08490cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02b79cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02b44c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08490a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08490a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02baac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0847720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08481b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084878a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa136bc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05832e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056b869>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0589235>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0589e87>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058c1fc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058c2de>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f0824>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa166578b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping)	Lustre: DEBUG MARKER: Started rundbench load pid=5683 ... LustreError: 4633:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 4633:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5503 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5450 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5614 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5551 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5652 to 0x280000400:5697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5715 to 0x240000400:5761) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5765 to 0x280000400:5793) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5829 to 0x240000400:5857) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 16115:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009885d800 ns: mdt-lustre-MDT0000_UUID lock: ffff88008f4b5c00/0xc841f08ceaacc6cb lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x116f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.47@tcp remote: 0x1d270f6ee1f7f671 expref: 3 pid: 16115 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: 3302:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737567838/real 1737567838] req@ffff88008bcae300 x1821968914669696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737567854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3302:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5917 to 0x240000400:5953) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5853 to 0x280000400:5889) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 3304:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880094050700 x1821968914801920/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6012 to 0x240000400:6049) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5948 to 0x280000400:5985) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6030 to 0x280000400:6049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6094 to 0x240000400:6113) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6125 to 0x280000400:6145) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6189 to 0x240000400:6209) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6281 to 0x240000400:6305) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6217 to 0x280000400:6241) Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 23413:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7251:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012bac0800 ns: mdt-lustre-MDT0001_UUID lock: ffff88009eb17000/0x61a57e510ac0f8f4 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0x67b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.34@tcp remote: 0xfef695c3b2493c96 expref: 4 pid: 7251 timeout: 0 lvb_type: 0 LustreError: 23413:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 23413 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02d9a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086b8a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0847f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08480cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02df9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02dc4c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08480a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08480a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02e2ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0846720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08471b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084778a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0efec4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05822e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056a869>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0588235>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0588e87>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058b1fc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058b2de>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08ef824>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11f978b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13521 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 7250:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7250:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 100 previous similar messages LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.203.134@tcp (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19939:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.134@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (19939) used greatest stack depth: 9792 bytes left Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:553 to 0x2c0000400:577) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:571 to 0x280000400:609) Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.34@tcp (stopping) Lustre: Skipped 3 previous similar messages	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 7758:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 5298:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012c864000 ns: mdt-lustre-MDT0000_UUID lock: ffff88009360e800/0xf45e52f1fcd0bcac lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1471:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.35@tcp remote: 0x80ad771da72e44b8 expref: 4 pid: 5298 timeout: 0 lvb_type: 0 LustreError: 7758:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 7758 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02b2a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08638a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa083ff87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08400cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02b89cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02b54c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08400a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08400a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02bbac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa083e720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa083f1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa083f78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa135bc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057a2e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0562869>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0580235>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0580e87>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05831fc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05832de>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08e7824>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa165578b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=6010 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 17 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 20 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5503 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5450 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 3295:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737505556/real 1737505556] req@ffff880093067100 x1821903329902336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737505572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3295:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5608 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5545 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5702 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5639 to 0x280000400:5729) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5807 to 0x280000400:5825) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5807 to 0x240000400:5825) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5893 to 0x240000400:5921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5893 to 0x280000400:5921) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 16 previous similar messages Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5986 to 0x240000400:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5986 to 0x280000400:6017) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6085 to 0x240000400:6113) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6084 to 0x280000400:6113) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6179 to 0x280000400:6209) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6179 to 0x240000400:6209) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 362:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 362:0:(osd_handler.c:698:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 17 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6275 to 0x240000400:6305) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6275 to 0x280000400:6305) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6369 to 0x280000400:6401) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6369 to 0x240000400:6401) Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 3mdts recovery; 1 clients	LustreError: 4920:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 10874:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e2d50958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802ed257840/0x8cf27d78f8a0450b lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1a9b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x8cf27d78f8a044fd expref: 3 pid: 10874 timeout: 0 lvb_type: 0 LustreError: 4920:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 4920 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01b7a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06e9139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa06c2f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06c30cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01bda1b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa06c30a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06c30a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01c0b16>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06c1720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06c21b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06c278a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f14c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03fb5a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03e3549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa04014f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0402147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa040438c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa0404470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa076bbf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa17539bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=30736 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031a495d28: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3770 to 0x2c0000402:3809) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3728 to 0x300000402:3745) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3729 to 0x340000402:3745) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3729 to 0x380000402:3745) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 31643:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:225) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:225) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:225) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:225) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -107 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -19 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -19 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -19 LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32170:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:3 to 0x340000400:33) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:33) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3987 to 0x340000402:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3987 to 0x300000402:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:4050 to 0x2c0000402:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3986 to 0x380000402:4001) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 808:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:257) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:257) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:257) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:257) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -107 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) Skipped 2 previous similar messages LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -19 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) Skipped 3 previous similar messages LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 1362:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:65) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:3 to 0x340000400:65) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 75 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:4231 to 0x380000402:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:4263 to 0x340000402:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:4263 to 0x300000402:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:4295 to 0x2c0000402:4321) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 23328:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 23328:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 191 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 2436:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:289) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:289) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:289) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:289) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002-lwp-OST0002: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 78 previous similar messages LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -19 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) Skipped 1 previous similar message LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:3 to 0x340000400:97) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:97) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:4486 to 0x380000402:4513) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:4518 to 0x340000402:4545) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:4518 to 0x300000402:4545) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:4550 to 0x2c0000402:4577) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 11 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4041:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.28@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: 4041:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:321) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:321) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:321) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 12 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) lustre-MDT0002-mdc-ffff88031a495d28: can't stat MDS #0: rc = -107 LustreError: 30761:0:(lmv_obd.c:1435:lmv_statfs()) Skipped 6 previous similar messages LDISKFS-fs (dm-2): recovery complete LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:3 to 0x340000400:129) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:129) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 13 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 31853:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7216:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8801243b4800 ns: mdt-lustre-MDT0001_UUID lock: ffff88012b45ba00/0xa5c08d1d7061221b lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x367:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.13@tcp remote: 0x42facacd1d5aca3a expref: 3 pid: 7216 timeout: 0 lvb_type: 0 LustreError: 31853:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 31853 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0219a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086c8a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0848f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08490cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa021f9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa021c4c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08490a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08490a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0222ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0847720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08481b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084878a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f08c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05832e2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa056b869>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa0589235>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0589e87>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058c1fc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058c2de>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f0824>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa120278b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)	Lustre: DEBUG MARKER: Started rundbench load pid=13400 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 7215:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7215:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 103 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19879:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.113@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:570 to 0x2c0000400:609) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24145:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.113@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1030 to 0x2c0000400:1057) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1029 to 0x280000400:1057) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28400:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.113@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1462 to 0x2c0000400:1505) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1461 to 0x280000400:1505) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5153) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5185) Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery	LustreError: 4654:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7262:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a4dd4800 ns: mdt-lustre-MDT0001_UUID lock: ffff880129a3ce00/0xf7e74cf3bffa0c97 lrc: 4/0,0 mode: PR/PR res: [0x2400036ca:0x1e07:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.39@tcp remote: 0xeb6a17affd9fbc63 expref: 3 pid: 7262 timeout: 0 lvb_type: 0 LustreError: 4654:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 4654 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) Lustre: Skipped 2 previous similar messages Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c2a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08998a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0875f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08760cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02c89cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02c54c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08760a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08760a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02cbac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0874720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08751b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa087578a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0ef1c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05b0382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0598899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05b62d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05b6f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05b929c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05b937e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa091d8b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11ec78b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: fail mds2 mds1 1 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 25273:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 23073:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012bc53800 ns: mdt-lustre-MDT0000_UUID lock: ffff880135495600/0x29b677bf58977220 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1269:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.16@tcp remote: 0xe6afc18004eca475 expref: 3 pid: 23073 timeout: 0 lvb_type: 0 LustreError: 25273:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 25273 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01a4a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa07e28a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa07bef87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa07bf0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01aa9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa01a74c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa07bf0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa07bf0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01adac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa07bd720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa07be1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa07be78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa134fc4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa049a382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0482899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa04a02d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa04a0f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa04a329c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa04a337e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08668b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa164980b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=5868 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: 3306:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737138236/real 1737138236] req@ffff8800b1b2a680 x1821518465800704/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737138252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3308:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1737138236/real 1737138236] req@ffff8800b1b2b100 x1821518465800832/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1737138252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3308:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 65 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5457 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5510 to 0x240000400:5537) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 8487:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 8487:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5559 to 0x280000400:5601) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5624 to 0x240000400:5665) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5669 to 0x280000400:5697) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5733 to 0x240000400:5761) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5761 to 0x280000400:5793) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5824 to 0x240000400:5857) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5926 to 0x240000400:5953) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5861 to 0x280000400:5889) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6016 to 0x240000400:6049) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5953 to 0x280000400:5985) Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 27600:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 24005:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009b6ba800 ns: mdt-lustre-MDT0001_UUID lock: ffff8800953d0200/0x27746508b81c3286 lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x36c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.39@tcp remote: 0xb7a182f2e48e86c expref: 3 pid: 24005 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0001: Not available for connect from 192.168.204.39@tcp (stopping) LustreError: 27600:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 27600 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0221a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08798a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0855f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08560cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02279cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02244c4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa08560a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08560a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa022aac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0854720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08551b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085578a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f30c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0590382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0578899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa05962d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0596f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa059929c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa059937e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08fd8b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa122a79b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=13400 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 38 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 9571:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 9571:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 100 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19881:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:587 to 0x280000400:609) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:570 to 0x2c0000400:609) Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.204.39@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24151:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.204.139@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:999 to 0x280000400:1025) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1000 to 0x2c0000400:1025) Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: 24005:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff88012f582a00 x1821313307522048/t0(0) o104->lustre-MDT0001@192.168.204.39@tcp:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 15023:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 13306:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802add953d8 ns: mdt-lustre-MDT0000_UUID lock: ffff880264558040/0xe4d724eae554eba lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x111c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xe4d724eae554eac expref: 2 pid: 13306 timeout: 0 lvb_type: 0 LustreError: 15023:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 15023 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0171a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0698139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0671be7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0671d2c>] ldlm_resource_clean+0x2c/0x50 [ptlrpc] [<ffffffffa0177a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0671d00>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0671d00>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa017ab26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0670640>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0671271>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa067184a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12a9c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03aa5a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0392549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03b04f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03b1147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03b338c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03b3470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa071abf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15709bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=25860 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802dc275d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 54 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 19 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.113@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3917 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3962 to 0x240000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3917 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3917 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3954 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3955 to 0x280000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3955 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4019 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 564:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 564:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802dc275d28: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4020 to 0x2c0000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4084 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3987 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4020 to 0x280000400:4065) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4057 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4089 to 0x280000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4153 to 0x240000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4090 to 0x2c0000400:4129) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4145 to 0x2c0000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4208 to 0x240000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4145 to 0x280000400:4161) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4113 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4181 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4148 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4181 to 0x2c0000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4245 to 0x240000400:4289) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 10776:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d3da1bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d5b4b880/0xe4d724eae54cb3b lrc: 3/0,0 mode: --/CW res: [0x20001a9e3:0xee4:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xe4d724eae54cb2d expref: 4 pid: 10776 timeout: 0 lvb_type: 0 Lustre: 17754:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736939056/real 1736939056] req@ffff880325a3bc40 x1821309619716736/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736939072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 17754:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 83 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4241 to 0x2c0000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4306 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4210 to 0x300000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4242 to 0x280000400:4257) Lustre: DEBUG MARKER: centos-111.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 24102:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 24102:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 10 PID: 24102 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0190a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0686139>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa065fbe7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa065fd2c>] ldlm_resource_clean+0x2c/0x50 [ptlrpc] [<ffffffffa0196a2b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa065fd00>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa065fd00>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0199b26>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa065e640>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa065f271>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa065f84a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12b6c4c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03985a2>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa0380549>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa039e4f5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa039f147>] class_process_config+0x547/0x26a0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a138c>] ? class_manual_cleanup+0xec/0x780 [obdclass] [<ffffffffa03a1470>] class_manual_cleanup+0x1d0/0x780 [obdclass] [<ffffffffa0708bf4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15d69bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=9403 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802eb1d2e98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 51 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3916 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3917 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3917 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 12985:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 12985:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4046 to 0x240000400:4065) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3950 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3950 to 0x280000400:3969) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802eb1d2e98: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4082 to 0x240000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3986 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3987 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4046 to 0x2c0000400:4065) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4047 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4014 to 0x300000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4111 to 0x240000400:4129) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4085 to 0x2c0000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4148 to 0x240000400:4193) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4053 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4085 to 0x280000400:4129) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 LustreError: 23823:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff880326120958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802c0be6940/0xc12a2effcc0d4ab6 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xee9:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xc12a2effcc0d4aa8 expref: 3 pid: 23823 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 19132:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 12002:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff880129f79800 ns: mdt-lustre-MDT0001_UUID lock: ffff88008e28c800/0x224d62705d74c337 lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x366:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.55@tcp remote: 0x81d15ef72c7f2726 expref: 3 pid: 12002 timeout: 0 lvb_type: 0 LustreError: 19132:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 19132 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02cea9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086cec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0849e57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0849f9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02d49cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0849f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0849f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02d7ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08485f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0849081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084965a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f04b8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05848c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa058a785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058b3d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058d74c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058d82e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f00c1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11c305b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13524 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 13627:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 13627:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 107 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 5838:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7215:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009a1ce000 ns: mdt-lustre-MDT0000_UUID lock: ffff8800938a9b00/0xc551388db5d61e91 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1a19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.201.25@tcp remote: 0xa7ae11e6fea88532 expref: 3 pid: 7215 timeout: 0 lvb_type: 0 LustreError: 5838:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 5838 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01eaa9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08d51f6>] ldlm_server_completion_ast+0x8a6/0x9b0 [ptlrpc] [<ffffffffa08adcc7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa08ade0f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01f09cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa08adde0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa08adde0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01f3ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08ac480>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08acef1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08ad4ca>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0efc9ec>] mdt_device_fini+0xdc/0x980 [mdt] [<ffffffffa05ec949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa05f27dd>] class_cleanup+0x23d/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05f3437>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05f57ac>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05f588e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa095243d>] server_put_super+0xa2d/0x1080 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11bc6bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) Lustre: Skipped 4 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13335 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: 7220:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7220:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 103 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages LustreError: 3769:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880092f22300 x1820835021712128/t0(0) o250->MGC192.168.201.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'ptlrpcd_00_01.0' uid:0 gid:0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5132 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5095 to 0x2c0000401:5121) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19885:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.125@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5510 to 0x280000401:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5478 to 0x2c0000401:5505) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24151:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.125@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (24151) used greatest stack depth: 9680 bytes left Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5864 to 0x2c0000401:5889) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5896 to 0x280000401:5921) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28404:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.125@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:513) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:513) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6250 to 0x2c0000401:6273) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6282 to 0x280000401:6305) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32656:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.125@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:545) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:545) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6614 to 0x2c0000401:6657) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6646 to 0x280000401:6689) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4509:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.125@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:577) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:577) Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 15681:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 15305:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f0f12e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f5891e40/0x42d3eab27213c570 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x148c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x42d3eab27213c562 expref: 3 pid: 15305 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 15681:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 11 PID: 15681 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa01bda9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06c58a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa06a1f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06a20cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01c39bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa01c04b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa06a20a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06a20a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa01c6ab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06a0720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a11b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a178a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f16c3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03dc382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03c4899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03e22d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03e2f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03e529c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03e537e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa07498b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa175c74b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=9377 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88031d01e678: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 66 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3929 to 0x300000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3929 to 0x2c0000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3929 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3973 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4030 to 0x240000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3998 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3998 to 0x2c0000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3998 to 0x280000400:4033) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4100 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4068 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4068 to 0x280000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4068 to 0x2c0000400:4097) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031d01e678: operation mds_reint to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.23@tcp (at 0@lo) Lustre: Skipped 34 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4124 to 0x300000400:4161) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4125 to 0x280000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4125 to 0x2c0000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4157 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4193 to 0x300000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4193 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4194 to 0x280000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4226 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4285 to 0x240000400:4321) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4253 to 0x300000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4253 to 0x280000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4253 to 0x2c0000400:4289) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 12542:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 12542:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.23@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4322 to 0x280000400:4353) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4322 to 0x300000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4354 to 0x240000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4322 to 0x2c0000400:4353) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: 13080:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031feb0958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802f80e52c0/0x42d3eab2720f91d1 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0x129f:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x42d3eab2720f91c3 expref: 4 pid: 13080 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4384 to 0x300000400:4417) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4415 to 0x240000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4383 to 0x2c0000400:4417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4383 to 0x280000400:4417) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4448 to 0x280000400:4481) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4448 to 0x300000400:4481) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4448 to 0x2c0000400:4481) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4480 to 0x240000400:4513) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 18993:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735773056/real 1735773056] req@ffff8802daefa840 x1820087073375488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735773072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 18993:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 78 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 18980:0:(client.c:3286:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800a28cc140 x1820087065815424/t317827580344(317827580344) o101->lustre-MDT0000-mdc-ffff88031d01e678@0@lo:12/10 lens 576/608 e 0 to 0 dl 1735773112 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 18980:0:(client.c:3286:ptlrpc_replay_interpret()) Skipped 444 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4545 to 0x240000400:4577) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4513 to 0x280000400:4545) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4514 to 0x2c0000400:4545) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4513 to 0x300000400:4545) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4578 to 0x280000400:4609) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4610 to 0x240000400:4641) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4578 to 0x2c0000400:4609) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4578 to 0x300000400:4609) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 12 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 23659:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 9574:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff8801369be000 ns: mdt-lustre-MDT0001_UUID lock: ffff88012e9afc00/0xd87986ffdb5b8775 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0x355:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.26@tcp remote: 0x5707ab4ddb24d05b expref: 4 pid: 9574 timeout: 0 lvb_type: 0 LustreError: 23659:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 23659 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa021da9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086bec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0848e57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0848f9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02239cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0848f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0848f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0226ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08475f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0848081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084865a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f04b8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05838c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0589785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058a3d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058c74c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058c82e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08ef0c1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11c305b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.26@tcp (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13705 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 7209:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7209:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 143 previous similar messages Lustre: Skipped 39 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.26@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 20203:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.126@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:557 to 0x280000400:577) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:541 to 0x2c0000400:577) Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg326-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.26@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18020:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 17647:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800af6fae98 ns: mdt-lustre-MDT0000_UUID lock: ffff88031cea0f40/0x2c47dc6121625cad lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x101a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2c47dc6121625c9f expref: 3 pid: 17647 timeout: 0 lvb_type: 0 LustreError: 18020:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 18020 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0209a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06c58a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa06a1f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06a20cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa020f9bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa020c4b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa06a20a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06a20a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0212ab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06a0720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06a11b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06a178a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0efcc3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03dc382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03c4899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03e22d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03e2f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03e529c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03e537e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa07498b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa173874b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=16718 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 14982:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e91a2e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802eb29b880/0x2c47dc612160a63f lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xeda:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x2c47dc612160a631 expref: 4 pid: 14982 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3922 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3922 to 0x280000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3966 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 17090:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff88031a57a548 ns: mdt-lustre-MDT0000_UUID lock: ffff88031c8d5a40/0x2c47dc61216185f9 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xefc:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2c47dc61216185eb expref: 3 pid: 17090 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.63@tcp (at 0@lo) Lustre: Skipped 24 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3967 to 0x280000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3967 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3967 to 0x2c0000400:4001) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4031 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-61.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8802f7ca8958: operation ldlm_cancel to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802f7ca8958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 29 previous similar messages	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 24447:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 21938:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009f884000 ns: mdt-lustre-MDT0000_UUID lock: ffff880094644e00/0xb0c953d625377121 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x11f2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.41@tcp remote: 0xf279e808a0218bdb expref: 3 pid: 21938 timeout: 0 lvb_type: 0 LustreError: 24447:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 24447 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa029fa9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa08b3ec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0890e57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0890f9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02a59cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0890f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0890f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02a8ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa088f5f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0890081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa089065a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa1359b8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05cb8c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa05d1785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05d23d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05d474c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05d482e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa09370b1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa161805b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=5880 ... LustreError: 5035:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5035:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 32 previous similar messages Lustre: 3296:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735479859/real 1735479859] req@ffff88009dfa0700 x1819779466432640/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735479875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3296:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5491 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5438 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 6137:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009f852800 ns: mdt-lustre-MDT0000_UUID lock: ffff88012e1b3a00/0xb0c953d625339aea lrc: 3/0,0 mode: --/PR res: [0x20001a9e3:0xee4:0x0].0x0 bits 0x13/0x8 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.41@tcp remote: 0xf279e808a0206cd4 expref: 3 pid: 6137 timeout: 0 lvb_type: 0 LustreError: 11540:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880094b34e00 x1819779466571392/t0(0) o105->lustre-MDT0000@192.168.204.41@tcp:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5537 to 0x280000400:5569) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5601 to 0x240000400:5633) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5695 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5630 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5782 to 0x240000400:5825) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5718 to 0x280000400:5761) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5895 to 0x240000400:5921) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5830 to 0x280000400:5857) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.41@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5913 to 0x280000400:5953) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5978 to 0x240000400:6017) Lustre: DEBUG MARKER: oleg441-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 18735:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 16167:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ee496fc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a37e9a80/0xa92bb5b15f260a8a lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xfae:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xa92bb5b15f260a7c expref: 3 pid: 16167 timeout: 0 lvb_type: 0 LustreError: 18735:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 18735 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0162a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a88a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0684f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06850cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01689bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa01654b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa06850a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06850a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa016bab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0683720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06841b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068478a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12a1c3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03bf382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03a7899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c52d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03c5f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03c829c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03c837e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa072c8b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15bd74b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=11360 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 32 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.98@tcp (at 0@lo) Lustre: Skipped 26 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3961 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3917 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3916 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.98@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages LustreError: 16166:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 16166:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: 32156:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735430478/real 1735430478] req@ffff880277ac4640 x1819727933482752/t0(0) o400->lustre-MDT0000-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735430494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 32156:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4017 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3953 to 0x300000400:3969) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3954 to 0x2c0000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3954 to 0x280000400:3969) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 18130:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 18130:0:(osd_handler.c:698:osd_ro()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 515:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 31305:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802aab32e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802af7aed00/0xa7c05697de14f4c8 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xef9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xa7c05697de14f4ba expref: 3 pid: 31305 timeout: 0 lvb_type: 0 LustreError: 515:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 12 PID: 515 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0192a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06a88a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0684f87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06850cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01989bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa01954b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa06850a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06850a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa019bab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0683720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06841b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa068478a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa129ac3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03bf382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03a7899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03c52d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03c5f27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03c829c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03c837e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa072c8b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa159474b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=11147 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8803192befc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 55 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 20 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.98@tcp (at 0@lo) Lustre: Skipped 57 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 2375:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735060208/real 1735060208] req@ffff8802c5988040 x1819339613162368/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735060224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 2375:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3954 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3952 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4010 to 0x240000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3954 to 0x2c0000400:3969) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 17165:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 17165:0:(osd_handler.c:698:osd_ro()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.98@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3988 to 0x2c0000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3988 to 0x300000400:4033) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3989 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4052 to 0x240000400:4097) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8803192befc8: operation mds_reint to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages LustreError: 2377:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802e9ea9940 x1819339614718976/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 LustreError: 2377:0:(client.c:1300:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4059 to 0x300000400:4097) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4059 to 0x280000400:4097) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4059 to 0x2c0000400:4097) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4123 to 0x240000400:4161) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4109 to 0x2c0000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4109 to 0x300000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4173 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 25993:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 25993:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 3 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4151 to 0x300000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4147 to 0x280000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4207 to 0x240000400:4225) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4153 to 0x2c0000400:4193) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4239 to 0x240000400:4257) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4206 to 0x2c0000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4207 to 0x280000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4206 to 0x300000400:4225) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 LustreError: 2374:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ac881e40 x1819339616564224/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4248 to 0x2c0000400:4289) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4248 to 0x280000400:4289) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4248 to 0x300000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4279 to 0x240000400:4321) Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 29239:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 28868:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f7a2a548 ns: mdt-lustre-MDT0000_UUID lock: ffff880320b1f0c0/0x5aa6554b4fd8e196 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0x1386:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5aa6554b4fd8e188 expref: 3 pid: 28868 timeout: 0 lvb_type: 0 LustreError: 29239:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 14 PID: 29239 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0213a9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06d18a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa06adf87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa06ae0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02199bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa02164b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa06ae0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa06ae0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa021cab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa06ac720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa06ad1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa06ad78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f05c3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03e8382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa03ee2d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03eef27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03f129c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03f137e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa07558b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa177e74b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=23496 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3921 to 0x240000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3964 to 0x2c0000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3921 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3920 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 LustreError: 23864:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800b71d37e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a18e4000/0x5aa6554b4fd066d2 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xfad:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x5aa6554b4fd066c4 expref: 4 pid: 23864 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4032 to 0x2c0000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3968 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3968 to 0x240000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3968 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff88031fdb8958: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection restored to 192.168.123.83@tcp (at 0@lo) Lustre: Skipped 29 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4028 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4028 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4092 to 0x2c0000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4028 to 0x240000400:4065) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4156 to 0x2c0000400:4193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4092 to 0x280000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4092 to 0x240000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4091 to 0x300000400:4129) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4155 to 0x300000400:4193) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4155 to 0x240000400:4193) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4219 to 0x2c0000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4154 to 0x280000400:4193) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.83@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 8 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4217 to 0x300000400:4257) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4218 to 0x280000400:4257) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4281 to 0x2c0000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4217 to 0x240000400:4257) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: 634:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1735016945/real 1735016945] req@ffff8800a1ade940 x1819294310794624/t0(0) o400->lustre-MDT0000-lwp-OST0002@0@lo:12/10 lens 224/224 e 0 to 1 dl 1735016961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 634:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 57 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4350 to 0x2c0000400:4385) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4286 to 0x300000400:4321) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4286 to 0x240000400:4321) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4286 to 0x280000400:4321) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4351 to 0x300000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4351 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4415 to 0x2c0000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4351 to 0x240000400:4385) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 617:0:(client.c:3286:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880323352840 x1819294306548096/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff88031fdb8958@0@lo:12/10 lens 576/608 e 0 to 0 dl 1735017052 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 617:0:(client.c:3286:ptlrpc_replay_interpret()) Skipped 417 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4416 to 0x300000400:4449) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4479 to 0x2c0000400:4513) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4415 to 0x280000400:4449) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4416 to 0x240000400:4449) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 28332:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f377b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a18e5a40/0x5aa6554b4fd7de9d lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0xef7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x5aa6554b4fd7de8f expref: 3 pid: 28332 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4478 to 0x280000400:4513) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4479 to 0x300000400:4513) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4478 to 0x240000400:4513) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4542 to 0x2c0000400:4577) Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 32435:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 30058:0:(ldlm_lockd.c:1492:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ab9a53d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802c5cc0400/0xe97cba38a91ddf8 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xefb:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xe97cba38a91ddea expref: 4 pid: 30058 timeout: 0 lvb_type: 0 LustreError: 32435:0:(ldlm_lockd.c:1064:ldlm_server_completion_ast()) LBUG CPU: 4 PID: 32435 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa017ea9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa06908a9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa066cf87>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa066d0cf>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01849bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa01814b4>] ? libcfs_debug_msg+0x6f4/0xc70 [libcfs] [<ffffffffa066d0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa066d0a0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0187ab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa066b720>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa066c1b1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa066c78a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12c8c3c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03a7382>] obd_precleanup+0x142/0x230 [obdclass] [<ffffffffa038f899>] ? class_disconnect_exports+0x119/0x310 [obdclass] [<ffffffffa03ad2d5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03adf27>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03b029c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03b037e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa07148b4>] server_put_super+0xb84/0x1210 [ptlrpc] [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15e474b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=24877 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-MDT0000-lwp-OST0002: Connection restored to 192.168.123.23@tcp (at 0@lo) Lustre: Skipped 53 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3915 to 0x2c0000400:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3958 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3915 to 0x300000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3914 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4018 to 0x240000400:4033) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3969 to 0x300000400:4001) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3964 to 0x2c0000400:4001) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3956 to 0x280000400:4001) Lustre: DEBUG MARKER: centos-21.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 6268:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 29289:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009ca5d800 ns: mdt-lustre-MDT0000_UUID lock: ffff880095395200/0x88c755a8ea8870af lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.45@tcp remote: 0x334c3e6f8ff04a7f expref: 3 pid: 29289 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) LustreError: 6268:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 6268 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02b9a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0865f29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0842eb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0842fff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02bf9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0842fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0842fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02c2ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0841650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08420e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08426ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa13495fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0580949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0586805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0587457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05897cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05898ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08e8dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa160a2ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 34 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=5912 ... LustreError: 5618:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5618:0:(osd_handler.c:698:osd_ro()) Skipped 13 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 3147:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2448:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff880131162800 ns: mdt-lustre-MDT0000_UUID lock: ffff88007db1c000/0xeb1b441b8a6e4e5d lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x14f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.54@tcp remote: 0xdb309abfb7d58acc expref: 3 pid: 2448 timeout: 0 lvb_type: 0 LustreError: 3147:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 3147 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02d2a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086eec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa084be57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084bf9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02d89cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084bf70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084bf70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02dbac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa084a5f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084b081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084b65a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f1fb8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa05868c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa058c785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058d3d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa058f74c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa058f82e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f20c1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11de05b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message	Lustre: DEBUG MARKER: Started rundbench load pid=10799 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5535 to 0x240000400:5569) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5482 to 0x280000400:5505) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5635 to 0x240000400:5665) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5571 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5731 to 0x240000400:5761) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5730 to 0x280000400:5761) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5827 to 0x240000400:5857) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5828 to 0x280000400:5857) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: 3496:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734625682/real 1734625682] req@ffff88012d433b80 x1818883410358656/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734625698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3496:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5919 to 0x280000400:5953) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5918 to 0x240000400:5953) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6024 to 0x280000400:6049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6025 to 0x240000400:6049) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6116 to 0x240000400:6145) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6116 to 0x280000400:6145) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6210 to 0x240000400:6241) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6210 to 0x280000400:6241) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6318 to 0x280000400:6337) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6319 to 0x240000400:6337) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-0): 2 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6430 to 0x240000400:6465) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6429 to 0x280000400:6465) Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 30372:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 28232:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800ac97d800 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a5cc8e00/0x7f8fba81662aceea lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x12eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.27@tcp remote: 0x4d7938f95b173ea9 expref: 3 pid: 28232 timeout: 0 lvb_type: 0 LustreError: 30372:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 30372 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa020ba9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0876ec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa0853e57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0853f9f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02119cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0853f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0853f70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0214ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08525f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0853081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa085365a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa1361b8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa058e8c9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0594785>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa05953d7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa059774c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa059782e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08fa201>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa162005b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=5816 ... LustreError: 4871:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 4871:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: 3294:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734593659/real 1734593659] req@ffff88012b470700 x1818850303297920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734593675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3294:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 63 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5502 to 0x240000400:5537) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5448 to 0x280000400:5473) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5608 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5545 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5701 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5638 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5723 to 0x280000400:5761) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5786 to 0x240000400:5825) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 LustreError: 16576:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a5f55000 ns: mdt-lustre-MDT0000_UUID lock: ffff88012feccc00/0x7f8fba816628080f lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xef9:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.204.27@tcp remote: 0x4d7938f95b166ec4 expref: 4 pid: 16576 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5827 to 0x280000400:5857) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5891 to 0x240000400:5921) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5974 to 0x240000400:6017) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5909 to 0x280000400:5953) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6013 to 0x280000400:6049) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6077 to 0x240000400:6113) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6103 to 0x280000400:6145) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:6167 to 0x240000400:6209) Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 21123:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 18538:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88008c936800 ns: mdt-lustre-MDT0000_UUID lock: ffff880135b6a200/0x6b389c1d7d90a57b lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0x1173:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.4@tcp remote: 0x5566b3f59751a52 expref: 3 pid: 18538 timeout: 0 lvb_type: 0 LustreError: 21123:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 21123 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0203a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0863f29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0840eb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0840fff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02099cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0840fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0840fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa020cac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa083f650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08400e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08406ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa134a5fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057e949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0584805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0585457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05877cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05878ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08e6dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa160b2ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=6441 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 19 previous similar messages Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5451 to 0x280000400:5473) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5505 to 0x240000400:5537) Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 10675:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 10675:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: 3308:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734462343/real 1734462343] req@ffff88008fdd9880 x1818711995187968/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734462359 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 3308:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.202.104@tcp (at 0@lo) Lustre: Skipped 12 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5612 to 0x240000400:5633) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5569) Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5711 to 0x240000400:5729) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5647 to 0x280000400:5665) Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5736 to 0x280000400:5761) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5799 to 0x240000400:5825) Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 6616:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 6616:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 0 PID: 6616 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa01fca9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa087f1f6>] ldlm_server_completion_ast+0x8a6/0x9b0 [ptlrpc] [<ffffffffa0857cc7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0857e0f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02029cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0857de0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0857de0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0205ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0856480>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0856ef1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08574ca>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa135b9ec>] mdt_device_fini+0xdc/0x980 [mdt] [<ffffffffa0595bc9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa059ba5d>] class_cleanup+0x23d/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa059c6b7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa059ea2c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa059eb0e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08fd81d>] server_put_super+0xa2d/0x1080 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa16206bb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.43@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=7101 ... LustreError: 5764:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 5764:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 27688:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff880129fe1800 ns: mdt-lustre-MDT0000_UUID lock: ffff88008e590900/0x858796c46644b8f3 lrc: 4/0,0 mode: CW/CW res: [0x20001a9e3:0xf03:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.43@tcp remote: 0x9fffce21715bdd1 expref: 4 pid: 27688 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 2373:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2373:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 2373 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa018aa9d>] lbug_with_loc+0x4d/0xa0 [libcfs] [<ffffffffa0681ec9>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81402511>] ? vsnprintf+0x201/0x6a0 [<ffffffffa065ee57>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa065efae>] ldlm_resource_clean+0x3e/0x50 [ptlrpc] [<ffffffffa01909bb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa065ef70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa065ef70>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0193ab6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa065d5f0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa065e081>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa065e65a>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa12a0b8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03998e9>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa039f7a5>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03a03f7>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03a276c>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03a284e>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa07050c1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa15baffb>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17	Lustre: DEBUG MARKER: Started rundbench load pid=7771 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 9939:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 9939:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 16 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.103@tcp (at 0@lo) Lustre: Skipped 52 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3959 to 0x240000400:4001) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3914 to 0x300000400:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3914 to 0x2c0000400:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3915 to 0x280000400:3937) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: MGC192.168.123.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 32 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3951 to 0x2c0000400:3969) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3951 to 0x300000400:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3951 to 0x280000400:3969) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4015 to 0x240000400:4033) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec LustreError: 13554:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: * setting device osd-zfs read-only * LustreError: 13554:0:(osd_handler.c:698:osd_ro()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 LustreError: lustre-MDT0000-mdc-ffff8800b00b37e8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3987 to 0x280000400:4033) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4051 to 0x240000400:4097) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:3987 to 0x300000400:4033) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3988 to 0x2c0000400:4033) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4048 to 0x280000400:4065) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4048 to 0x300000400:4065) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4111 to 0x240000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4048 to 0x2c0000400:4065) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4083 to 0x280000400:4129) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4084 to 0x300000400:4129) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4084 to 0x2c0000400:4129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4148 to 0x240000400:4193) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 6 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4142 to 0x300000400:4161) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4143 to 0x2c0000400:4161) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4207 to 0x240000400:4225) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4143 to 0x280000400:4161) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: 28980:0:(client.c:2364:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1734294831/real 1734294831] req@ffff8802de767340 x1818536991045120/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1734294847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 28980:0:(client.c:2364:ptlrpc_expire_one_request()) Skipped 80 previous similar messages Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 LustreError: 28976:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802e0912840 x1818536991425152/t0(0) o6->lustre-OST0003-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-3-0.0' uid:0 gid:0 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 28976:0:(client.c:1300:ptlrpc_import_delay_req()) Skipped 11 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4179 to 0x280000400:4225) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4244 to 0x240000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4180 to 0x2c0000400:4225) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4180 to 0x300000400:4225) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 8 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4244 to 0x300000400:4289) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4243 to 0x2c0000400:4289) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4307 to 0x240000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4243 to 0x280000400:4289) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 28969:0:(client.c:3286:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8803222b7840 x1818536988534016/t317827580343(317827580343) o101->lustre-MDT0000-mdc-ffff8800b00b37e8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1734294937 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 28969:0:(client.c:3286:ptlrpc_replay_interpret()) Skipped 257 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4308 to 0x300000400:4353) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4307 to 0x280000400:4353) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4308 to 0x2c0000400:4353) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4371 to 0x240000400:4417) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000400:4365 to 0x300000400:4385) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4366 to 0x2c0000400:4385) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4366 to 0x280000400:4385) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:4430 to 0x240000400:4449) Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 LustreError: 455:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ae8db7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802c8bdf0c0/0x79c767845765aed7 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf0f:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0x79c767845765aec9 expref: 2 pid: 455 timeout: 0 lvb_type: 0	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 19295:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 2075:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff880099a94800 ns: mdt-lustre-MDT0001_UUID lock: ffff8800973fe000/0x819a797408928648 lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0x366:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.201.15@tcp remote: 0xbe1659847164d799 expref: 3 pid: 2075 timeout: 0 lvb_type: 0 LustreError: 19295:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 19295 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa0226a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0872f29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa084feb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084ffff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa022c9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084ffd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084ffd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa022fac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa084e650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084f0e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084f6ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0ef85fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa058d949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0593805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0594457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05967cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05968ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f5dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11b92ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping)	Lustre: DEBUG MARKER: Started rundbench load pid=13759 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages LustreError: 7264:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7264:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 126 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 37 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping)	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 3781:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 16727:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88008818f000 ns: mdt-lustre-MDT0001_UUID lock: ffff8800a45b8c00/0x25ce0894597affe7 lrc: 4/0,0 mode: PR/PR res: [0x2400007ed:0xe5d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.21@tcp remote: 0xdf7040506f9cc44f expref: 3 pid: 16727 timeout: 0 lvb_type: 0 LustreError: 3781:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 3781 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02c8a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa085cf29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa0839eb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa0839fff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02ce9cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0839fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0839fd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02d1ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0838650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa08390e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08396ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f0ed8c>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0577949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa057d805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa057e457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05807cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05808ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08dfdc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11ce2ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13561 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages LustreError: 8382:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 8382:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 107 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.202.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.202.121@tcp (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19949:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.121@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:570 to 0x2c0000400:609) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:586 to 0x280000400:609) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24220:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.121@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:921 to 0x2c0000400:961) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:920 to 0x280000400:961) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28474:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.121@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1313 to 0x280000400:1345) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1312 to 0x2c0000400:1345) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5185) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32730:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.121@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1677 to 0x2c0000400:1697) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1676 to 0x280000400:1697) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg221-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5153) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5217) Lustre: DEBUG MARKER: oleg221-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.21@tcp (stopping)	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 23414:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7214:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88008fa1e000 ns: mdt-lustre-MDT0001_UUID lock: ffff88006d98e600/0x2ee88dbc909de82 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0x796:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.201.40@tcp remote: 0x96a36ac77f53812a expref: 4 pid: 7214 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0001: Not available for connect from 192.168.201.40@tcp (stopping) LustreError: 23414:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 2 PID: 23414 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa021ca9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086ef29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa084beb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084bfff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02229cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa084bfd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa084bfd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0225ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa084a650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa084b0e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa084b6ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f085fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa0589949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa058f805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0590457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05927cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05928ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08f1dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11c931b>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 192.168.201.40@tcp (stopping) Lustre: Skipped 2 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13587 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 20698:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 20698:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 126 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19946:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.201.140@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:570 to 0x2c0000400:609) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609) Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg140-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg140-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 8067:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7206:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff88012ff57000 ns: mdt-lustre-MDT0001_UUID lock: ffff880091878a00/0xa7116b1602c9c9a4 lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0x36c:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.51@tcp remote: 0xf4166e9b63e8696 expref: 4 pid: 7206 timeout: 0 lvb_type: 0 LustreError: 8067:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 3 PID: 8067 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02cca9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa0861f29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffff81401591>] ? vsnprintf+0x201/0x6a0 [<ffffffffa083eeb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa083efff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02d29cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa083efd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa083efd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02d5ac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa083d650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa083e0e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa083e6ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0ef95fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa057c949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa0582805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa0583457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05857cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05858ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08e4dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11ba2ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13594 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 20688:0:(ldlm_lib.c:1094:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 20688:0:(ldlm_lib.c:1094:target_handle_connect()) Skipped 117 previous similar messages Lustre: Skipped 25 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) Lustre: Skipped 25 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 19970:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.151@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:802 to 0x2c0000400:833) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:818 to 0x280000400:833) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 24236:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.151@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1253 to 0x280000400:1281) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1254 to 0x2c0000400:1281) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 28487:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.151@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1667 to 0x2c0000400:1761) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1667 to 0x280000400:1697) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5185) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5121) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 32735:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.151@tcp: no remote llog for lustre-sptlrpc, check MGS config mount.lustre (32735) used greatest stack depth: 9808 bytes left Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2106 to 0x280000400:2145) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2168 to 0x2c0000400:2209) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5217) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5153) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 4591:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.203.151@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2548 to 0x280000400:2593) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2612 to 0x2c0000400:2657) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 11 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5249) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5185) Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 12 times Lustre: Failing over lustre-MDT0001	Link to test
replay-single test 70b: dbench 2mdts recovery; 1 clients	LustreError: 3855:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 7245:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff880092d20800 ns: mdt-lustre-MDT0001_UUID lock: ffff88013638dd40/0x40d148147c52fc5d lrc: 4/0,0 mode: CW/CW res: [0x2400007ed:0xcb3:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.202.46@tcp remote: 0xd532757bad821025 expref: 4 pid: 7245 timeout: 0 lvb_type: 0 LustreError: 3855:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 1 PID: 3855 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 Call Trace: [<ffffffff817ded29>] dump_stack+0x19/0x1b [<ffffffffa02d1a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa086d086>] ldlm_server_completion_ast+0x8a6/0x9b0 [ptlrpc] [<ffffffffa0845747>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa084588f>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa02d797b>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa0845860>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa0845860>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa02daa76>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa08443d0>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa0844dd1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa08453aa>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa0f1361c>] mdt_device_fini+0xdc/0x980 [mdt] [<ffffffffa0588469>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa058e2fd>] class_cleanup+0x23d/0x950 [obdclass] [<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa058ef57>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff81220310>] ? __kmalloc+0x1e0/0x340 [<ffffffffa05912cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa05913ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa08e7bed>] server_put_super+0xa2d/0x1080 [ptlrpc] [<ffffffff817e8d7e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff812476ca>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81247ac2>] kill_anon_super+0x12/0x20 [<ffffffffa11cfaab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81247ec9>] deactivate_locked_super+0x49/0x60 [<ffffffff81248616>] deactivate_super+0x46/0x60 [<ffffffff81268b1f>] cleanup_mnt+0x3f/0x80 [<ffffffff81268bb2>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817f4363>] int_signal+0x12/0x17 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) Lustre: Skipped 3 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=13354 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 128 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 34 previous similar messages LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: Skipped 41 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 5 previous similar messages mount.lustre (20029) used greatest stack depth: 9840 bytes left Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:552 to 0x2c0000400:577) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:570 to 0x280000400:609) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 3 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 4 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping) Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:832 to 0x2c0000400:865) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:865 to 0x280000400:897) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 5 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 6 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1116 to 0x2c0000400:1153) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1148 to 0x280000400:1185) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5121) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5185) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-1): recovery complete LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1447 to 0x280000400:1473) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1415 to 0x2c0000400:1441) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 9 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5217) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5153) Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 10 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 192.168.202.46@tcp (stopping)	Link to test
replay-single test 70b: dbench 1mdts recovery; 1 clients	LustreError: 28042:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed: LustreError: 22049:0:(ldlm_lockd.c:1478:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802f3780958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d522da40/0xfd9b7e8996b10fe9 lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xeff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xfd9b7e8996b10fdb expref: 3 pid: 22049 timeout: 0 lvb_type: 0 LustreError: 28042:0:(ldlm_lockd.c:1050:ldlm_server_completion_ast()) LBUG CPU: 7 PID: 28042 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Call Trace: [<ffffffff817d93f8>] dump_stack+0x19/0x1b [<ffffffffa0193a9d>] lbug_with_loc+0x4d/0xb0 [libcfs] [<ffffffffa06bdf29>] ldlm_server_completion_ast+0x8a9/0x9b0 [ptlrpc] [<ffffffffa069aeb7>] cleanup_resource+0x1e7/0x300 [ptlrpc] [<ffffffffa069afff>] ldlm_resource_clean+0x2f/0x50 [ptlrpc] [<ffffffffa01999cb>] cfs_hash_for_each_relax+0x23b/0x450 [libcfs] [<ffffffffa069afd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa069afd0>] ? cleanup_resource+0x300/0x300 [ptlrpc] [<ffffffffa019cac6>] cfs_hash_for_each_nolock+0x76/0x200 [libcfs] [<ffffffffa0699650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc] [<ffffffffa069a0e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa069a6ba>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc] [<ffffffffa128c5fc>] mdt_device_fini+0xdc/0x9a0 [mdt] [<ffffffffa03d8949>] obd_precleanup+0x1d9/0x230 [obdclass] [<ffffffffa03de805>] class_cleanup+0x245/0x950 [obdclass] [<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90 [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffffa03df457>] class_process_config+0x547/0x27d0 [obdclass] [<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370 [<ffffffffa03e17cc>] ? class_manual_cleanup+0xec/0x770 [obdclass] [<ffffffffa03e18ae>] class_manual_cleanup+0x1ce/0x770 [obdclass] [<ffffffffa0740dc1>] server_put_super+0xa21/0x1070 [ptlrpc] [<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20 [<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0 [<ffffffff81248a62>] kill_anon_super+0x12/0x20 [<ffffffffa156f2ab>] lustre_kill_super+0x2b/0x30 [lustre] [<ffffffff81248e69>] deactivate_locked_super+0x49/0x60 [<ffffffff812495b6>] deactivate_super+0x46/0x60 [<ffffffff81269abf>] cleanup_mnt+0x3f/0x80 [<ffffffff81269b52>] __cleanup_mnt+0x12/0x20 [<ffffffff810b69b5>] task_work_run+0xb5/0xf0 [<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0 [<ffffffff817ee363>] int_signal+0x12/0x17 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages	Lustre: DEBUG MARKER: Started rundbench load pid=25857 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff8802f41c6fc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 52 previous similar messages	Link to test

Failing Test

Full Crash

Messages before crash

Comment

replay-single test 70b: dbench 1mdts recovery; 1 clients

LustreError: 25492:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) ASSERTION( data != ((void *)0) ) failed:
LustreError: 25492:0:(ldlm_lockd.c:1019:ldlm_server_completion_ast()) LBUG
CPU: 9 PID: 25492 Comm: umount Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
Call Trace:
[<ffffffff817d93f8>] dump_stack+0x19/0x1b
[<ffffffffa016eb4d>] lbug_with_loc+0x4d/0xa0 [libcfs]
[<ffffffffa069b229>] ldlm_server_completion_ast+0x8b9/0x9d0 [ptlrpc]
[<ffffffffa0677eb7>] cleanup_resource+0x1e7/0x320 [ptlrpc]
[<ffffffffa067802e>] ldlm_resource_clean+0x3e/0x50 [ptlrpc]
[<ffffffffa0380ddb>] cfs_hash_for_each_relax+0x23b/0x450 [obdclass]
[<ffffffffa0677ff0>] ? cleanup_resource+0x320/0x320 [ptlrpc]
[<ffffffffa0677ff0>] ? cleanup_resource+0x320/0x320 [ptlrpc]
[<ffffffffa0383f96>] cfs_hash_for_each_nolock+0x76/0x200 [obdclass]
[<ffffffffa0676650>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc]
[<ffffffffa06770e1>] __ldlm_namespace_free+0x61/0x5e0 [ptlrpc]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffffa06776bb>] ldlm_namespace_free_prior+0x5b/0x210 [ptlrpc]
[<ffffffffa12d2b5c>] mdt_device_fini+0xdc/0x8f0 [mdt]
[<ffffffffa0392d53>] obd_precleanup+0x163/0x290 [obdclass]
[<ffffffffa037d919>] ? class_disconnect_exports+0x119/0x310 [obdclass]
[<ffffffffa0398e65>] class_cleanup+0x245/0x860 [obdclass]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffffa03999c7>] class_process_config+0x547/0x2660 [obdclass]
[<ffffffff8122054f>] ? __kmalloc+0x1ef/0x370
[<ffffffffa039bbcc>] ? class_manual_cleanup+0xec/0x780 [obdclass]
[<ffffffffa039bcb0>] class_manual_cleanup+0x1d0/0x780 [obdclass]
[<ffffffffa0726234>] server_put_super+0xb84/0x1220 [ptlrpc]
[<ffffffff8124866a>] generic_shutdown_super+0x6a/0xf0
[<ffffffff81248a62>] kill_anon_super+0x12/0x20
[<ffffffffa15f311b>] lustre_kill_super+0x2b/0x30 [lustre]
[<ffffffff81248e69>] deactivate_locked_super+0x49/0x60
[<ffffffff812495b6>] deactivate_super+0x46/0x60
[<ffffffff81269abf>] cleanup_mnt+0x3f/0x80
[<ffffffff81269b52>] __cleanup_mnt+0x12/0x20
[<ffffffff810b69b5>] task_work_run+0xb5/0xf0
[<ffffffff8102ccb2>] do_notify_resume+0x92/0xb0
[<ffffffff817ee363>] int_signal+0x12/0x17

Lustre: DEBUG MARKER: Started rundbench load pid=24171 ...
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: test_70b fail mds1 1 times
Lustre: Failing over lustre-MDT0000
Lustre: lustre-MDT0000-mdc-ffff8803265f1bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 55 previous similar messages
LustreError: 19879:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802aa25a548 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a8aff0c0/0xfa74e1f72e8d207f lrc: 4/0,0 mode: PR/PR res: [0x20001a9e3:0xf1b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50306400000000 nid: 0@lo remote: 0xfa74e1f72e8d2071 expref: 2 pid: 19879 timeout: 0 lvb_type: 0