LustreError: 11715:0:(llog_osd.c:628:llog_osd_write_rec()) LBUG
Pid: 11715, comm: mdt07_013 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] libcfs_call_trace+0x90/0xf0 [libcfs]
[<0>] lbug_with_loc+0x4c/0xa0 [libcfs]
[<0>] llog_osd_write_rec+0x1a85/0x1ac0 [obdclass]
[<0>] llog_write_rec+0x290/0x590 [obdclass]
[<0>] llog_cat_add_rec+0x1e1/0x990 [obdclass]
[<0>] llog_add+0x17f/0x1f0 [obdclass]
[<0>] sub_updates_write+0x303/0xe32 [ptlrpc]
[<0>] top_trans_stop+0x4a2/0xfb0 [ptlrpc]
[<0>] lod_trans_stop+0x25c/0x340 [lod]
[<0>] mdd_trans_stop+0x28/0x16e [mdd]
[<0>] mdd_xattr_del+0x239/0x670 [mdd]
[<0>] mdt_reint_setxattr+0xc9f/0x1210 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb50 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe

LustreError: 11070:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x240000403:0x17:0x0] mdc close failed: rc = -116
Lustre: 7185:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802602c70c0 x1729879002943040/t4294967874(0) o101->25b6622e-f7b0-41a7-afdd-bcb2a74f3477@0@lo:643/0 lens 376/840 e 0 to 0 dl 1649741328 ref 1 fl Interpret:H/2/0 rc 0/0 job:'dd.0'
LustreError: 11160:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d824a88: inode [0x240000404:0x53:0x0] mdc close failed: rc = -116
LustreError: 11160:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 31961:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88005d4e0cc0 x1729879003102208/t4294967809(0) o101->e69a90e6-91b0-4a42-8c91-e9bbac9fbe7a@0@lo:649/0 lens 376/816 e 0 to 0 dl 1649741334 ref 1 fl Interpret:H/2/0 rc 0/0 job:'dd.0'
LustreError: 14561:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x240000404:0x1:0x0] mdc close failed: rc = -116
LustreError: 31968:0:(mdd_object.c:3465:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x42:0x0]: rc = -2
LustreError: 13262:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d824a88: inode [0x200000403:0x42:0x0] mdc close failed: rc = -2
Lustre: 9912:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e8933240 x1729879003796224/t4294968932(0) o101->25b6622e-f7b0-41a7-afdd-bcb2a74f3477@0@lo:663/0 lens 376/840 e 0 to 0 dl 1649741348 ref 1 fl Interpret:H/2/0 rc 0/0 job:'dd.0'
LustreError: 21358:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x240000403:0xbb:0x0] mdc close failed: rc = -116
Lustre: mdt00_006: service thread pid 8760 was inactive for 40.009 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 8760, comm: mdt00_006 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0x12e/0x2c50 [mdt]
[<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 9379:0:(mdt_handler.c:754:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0xee:0x0] ACL: rc = -2
LustreError: 23761:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x240000404:0xad:0x0] mdc close failed: rc = -116
LustreError: 9978:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0000: '9' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 9' to finish migration.
Lustre: dir [0x240000403:0xb2:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: mdt07_004: service thread pid 8633 was inactive for 62.067 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 8633, comm: mdt07_004 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_object_find_lock+0x6a/0x1a0 [mdt]
[<0>] mdt_reint_setxattr+0x342/0x1210 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb50 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 31950, comm: mdt04_000 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0xdb6/0x2c50 [mdt]
[<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt01_006: service thread pid 9030 was inactive for 62.157 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt05_005: service thread pid 9053 was inactive for 62.075 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
LustreError: 551:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 551:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 8699:0:(llog_cat.c:602:llog_cat_add_rec()) llog_write_rec -116: lh=ffff88025f00b858
LustreError: 8699:0:(update_trans.c:1062:top_trans_stop()) lustre-MDT0000-osp-MDT0001: write updates failed: rc = -116
LustreError: 21560:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -116
LustreError: 551:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 551:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 551:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 551:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 7885:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d824a88: inode [0x200000403:0x119:0x0] mdc close failed: rc = -116
Lustre: dir [0x240000403:0x24b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880243dd9e40/0xb56a1a02826c6563 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x6:0x0].0x0 bits 0x12/0x0 rrc: 16 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a02826c613b expref: 39 pid: 31938 timeout: 18156 lvb_type: 0
LustreError: 31846:0:(mdd_object.c:3465:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x42:0x0]: rc = -2
LustreError: 31938:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88025838c138 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a783bc40/0xb56a1a02826c8839 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x6:0x0].0x0 bits 0x12/0x0 rrc: 13 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb56a1a02826c7cc4 expref: 22 pid: 31938 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88022f5bae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88022f5bae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 8311:0:(file.c:5115:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x6:0x0] error: rc = -5
LustreError: 11654:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88022f5bae98: namespace resource [0x200000400:0x4:0x0].0x0 (ffff88031ea99940) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
LustreError: 14867:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 3' to finish migration.
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000403:0xa5:0x0]/0xa): rc = 0
Lustre: 6788:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x4d:0x0] with magic=0xbd60bd0
Lustre: 9019:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x57:0x0] with magic=0xbd60bd0
Lustre: 9019:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 31938:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x27:0x0] with magic=0xbd60bd0
Lustre: 31938:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 25798:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d824a88: inode [0x200000403:0x177:0x0] mdc close failed: rc = -2
LustreError: 25798:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 15057:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0000: '1' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 1' to finish migration.
Lustre: mdt03_003: service thread pid 3912 was inactive for 62.114 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: mdt02_002: service thread pid 31946 was inactive for 66.116 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: dir [0x200000403:0x191:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000403:0x191:0x0]/0xa): rc = 0
LustreError: 15293:0:(osd_index.c:217:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -2
LustreError: 18697:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0000: '11' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 11' to finish migration.
LustreError: 31951:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0001: '2' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 2' to finish migration.
LustreError: 31951:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8800a7838040/0xb56a1a028271c17f lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x6:0x0].0x0 bits 0x12/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a02826c7de3 expref: 128 pid: 15057 timeout: 18256 lvb_type: 0
LustreError: 14887:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880290dd6678 ns: mdt-lustre-MDT0000_UUID lock: ffff880284090040/0xb56a1a028271d22d lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x6:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb56a1a028271c57d expref: 24 pid: 14887 timeout: 0 lvb_type: 0
LustreError: 14887:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 7 previous similar messages
LustreError: 11-0: lustre-MDT0000-mdc-ffff88022f5bae98: operation mds_reint to node 0@lo failed: rc = -107
LustreError: Skipped 7 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88022f5bae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 11477:0:(file.c:5115:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0x12:0x0] error: rc = -5
LustreError: 11477:0:(file.c:5115:ll_inode_revalidate_fini()) Skipped 38 previous similar messages
LustreError: 20117:0:(mdc_request.c:1474:mdc_read_page()) lustre-MDT0000-mdc-ffff88022f5bae98: [0x200000403:0x1:0x0] lock enqueue fails: rc = -5
LustreError: 20117:0:(mdc_request.c:1474:mdc_read_page()) Skipped 1 previous similar message
LustreError: 24183:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88022f5bae98: namespace resource [0x200000404:0x12:0x0].0x0 (ffff88028606e840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 24183:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 21 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
LustreError: 20664:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x280000403:0x225:0x0] mdc close failed: rc = -116
LustreError: 20664:0:(file.c:243:ll_close_inode_openhandle()) Skipped 13 previous similar messages
Lustre: 26268:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802911be440 x1729879013188096/t0(0) o10->lustre-OST0001-osc-ffff88028d824a88@0@lo:6/4 lens 440/432 e 0 to 0 dl 1649741552 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
1[7372]: segfault at 8 ip 00007ff7e64c67e8 sp 00007ffc80e683d0 error 4 in ld-2.17.so[7ff7e64bb000+22000]
LustreError: 7820:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0001: '17' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 17' to finish migration.
Lustre: dir [0x280000404:0xbc8:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
9[24304]: segfault at 8 ip 00007f3e100177e8 sp 00007fff2bfbc670 error 4 in ld-2.17.so[7f3e1000c000+22000]
LustreError: 8566:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 0' to finish migration.
LustreError: 7808:0:(llite_nfs.c:338:ll_dir_get_parent_fid()) lustre: failure inode [0x280000404:0x68f:0x0] get parent: rc = -2
Lustre: dir [0x240000404:0x686:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000403:0x3be:0x0]/0xa): rc = 0
Lustre: 8699:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x3e9:0x0] with magic=0xbd60bd0
Lustre: 8699:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff880295c39300/0xb56a1a02827baa7a lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x50c:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a02827baa26 expref: 152 pid: 9019 timeout: 18365 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff88028d824a88: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff88028d824a88: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff88028d824a88: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 31839:0:(ldlm_lockd.c:2500:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1649741586 with bad export cookie 13072289466476663552
LustreError: 4374:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 27839:0:(file.c:5115:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x50c:0x0] error: rc = -5
LustreError: 27839:0:(file.c:5115:ll_inode_revalidate_fini()) Skipped 8 previous similar messages
LustreError: 1966:0:(mdc_request.c:1474:mdc_read_page()) lustre-MDT0001-mdc-ffff88028d824a88: [0x240000403:0x569:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0001-mdc-ffff88028d824a88: Connection restored to 192.168.123.107@tcp (at 0@lo)
14[10466]: segfault at 8 ip 00007f58976667e8 sp 00007ffd4696fd00 error 4 in ld-2.17.so[7f589765b000+22000]
LustreError: 11975:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0001: '19' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 19' to finish migration.
LustreError: 11975:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
8[12793]: segfault at 8 ip 00007fa0a285f7e8 sp 00007fff0209fd30 error 4 in ld-2.17.so[7fa0a2854000+22000]
Lustre: 26274:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88027ec53ec0 x1729879018771136/t0(0) o10->lustre-OST0000-osc-ffff88028d824a88@0@lo:6/4 lens 440/432 e 0 to 0 dl 1649741655 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 17113:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d824a88: inode [0x240000404:0x719:0x0] mdc close failed: rc = -2
LustreError: 17113:0:(file.c:243:ll_close_inode_openhandle()) Skipped 20 previous similar messages
LustreError: 16580:0:(mdd_object.c:3465:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x5fe:0x0]: rc = -2
LustreError: 16580:0:(mdd_object.c:3465:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x5fe:0x0]: rc = -2
Lustre: dir [0x200000403:0x60b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
LustreError: 25365:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0000: '11' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 11' to finish migration.
LustreError: 25365:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880243dda200/0xb56a1a0282922b11 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x5e7:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a0282922af5 expref: 136 pid: 2673 timeout: 18600 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88022f5bae98: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88022f5bae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 21444:0:(llite_lib.c:3119:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 21803:0:(file.c:5115:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -108
LustreError: 21803:0:(file.c:5115:ll_inode_revalidate_fini()) Skipped 1 previous similar message
LustreError: 28354:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88022f5bae98: namespace resource [0x200000403:0x5e7:0x0].0x0 (ffff880293694040) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 28354:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
17[27582]: segfault at 8 ip 00007f4e0b9377e8 sp 00007ffd73ead0a0 error 4 in ld-2.17.so[7f4e0b92c000+22000]
LustreError: 32622:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) lustre-MDD0002: '13' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 13' to finish migration.
LustreError: 32622:0:(mdd_dir.c:4213:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802822c1300/0xb56a1a02829345c6 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x914:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a02829345aa expref: 216 pid: 31938 timeout: 18612 lvb_type: 0
LustreError: 11-0: lustre-MDT0002-mdc-ffff88028d824a88: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff88028d824a88: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0002-mdc-ffff88028d824a88: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 3005:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 3365:0:(mdc_request.c:1474:mdc_read_page()) lustre-MDT0002-mdc-ffff88028d824a88: [0x280000400:0x2e:0x0] lock enqueue fails: rc = -108
LustreError: 3365:0:(mdc_request.c:1474:mdc_read_page()) Skipped 2 previous similar messages
Lustre: dir [0x280000404:0xf6d:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
LustreError: 26268:0:(mdc_locks.c:1389:mdc_intent_getattr_async_interpret()) lustre-MDT0002-mdc-ffff88028d824a88: ldlm_cli_enqueue_fini() failed: rc = -5
LustreError: 333:0:(file.c:5115:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x914:0x0] error: rc = -5
LustreError: 333:0:(file.c:5115:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
Lustre: lustre-MDT0002-mdc-ffff88028d824a88: Connection restored to 192.168.123.107@tcp (at 0@lo)
LustreError: 32138:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88008ab8e678 ns: mdt-lustre-MDT0002_UUID lock: ffff8802f4a1d2c0/0xb56a1a0282978972 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x896:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb56a1a0282976b0a expref: 12 pid: 32138 timeout: 0 lvb_type: 0
LustreError: 32138:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 6 previous similar messages
Lustre: 18542:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x10f3:0x0] with magic=0xbd60bd0
Lustre: 18542:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 17709:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88022f5bae98: inode [0x280000403:0x8db:0x0] mdc close failed: rc = -13
LustreError: 17709:0:(file.c:243:ll_close_inode_openhandle()) Skipped 38 previous similar messages
7[26150]: segfault at 8 ip 00007f265d7127e8 sp 00007fff8b44de70 error 4 in ld-2.17.so[7f265d707000+22000]
LustreError: 1495:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 1495:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 1495:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 1495:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: dir [0x240000404:0xe05:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 11689:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0xeba:0x0] with magic=0xbd60bd0
Lustre: 11689:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 31966:0:(mdd_orphans.c:282:mdd_orphan_delete()) lustre-MDD0001: could not delete orphan object [0x240000404:0xe4e:0x0]: rc = -2
LustreError: 31966:0:(mdd_object.c:3517:mdd_close()) lustre-MDD0001: unable to delete [0x240000404:0xe4e:0x0] from orphan list: rc = -2
LustreError: 8566:0:(mdd_object.c:401:mdd_xattr_get()) lustre-MDD0002: object [0x280000405:0x30c:0x0] not found: rc = -2
LustreError: 31846:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88009173de00/0xb56a1a02829f56e1 lrc: 3/0,0 mode: PR/PR res: [0x200000407:0x11e:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb56a1a02829f56d3 expref: 195 pid: 31960 timeout: 18803 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88022f5bae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88022f5bae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 30750:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 30750:0:(llite_lib.c:1871:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 31131:0:(llite_lib.c:3119:ll_prep_inode()) new_inode -fatal: rc -2
Lustre: lustre-MDT0000-mdc-ffff88022f5bae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
LustreError: 9437:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880274232e98 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cc2b5680/0xb56a1a0282a9d918 lrc: 3/0,0 mode: PR/PR res: [0x200000407:0x314:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb56a1a0282a9d8f5 expref: 4 pid: 9437 timeout: 0 lvb_type: 0
LustreError: 9437:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 2 previous similar messages
LustreError: 26435:0:(llite_lib.c:1708:ll_update_lsm_md()) lustre: [0x280000405:0x1c6:0x0] dir layout mismatch:
LustreError: 26435:0:(lustre_lmv.h:142:lsm_md_dump()) magic 0xcd20cd0 stripe count 1 master mdt 2 hash type crush:0x2000003 max-inherit 0 max-inherit-rr 0 version 1 migrate offset 0 migrate hash 0x0 pool
LustreError: 26435:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[0] [0x280000400:0x3a:0x0]
LustreError: 26435:0:(lustre_lmv.h:142:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 2 hash type crush:0x82000003 max-inherit 0 max-inherit-rr 0 version 0 migrate offset 1 migrate hash 0x2 pool
LustreError: 26435:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[0] [0x280000400:0x3a:0x0]
LustreError: 26433:0:(lustre_lmv.h:142:lsm_md_dump()) magic 0xcd20cd0 stripe count 1 master mdt 2 hash type crush:0x2000003 max-inherit 0 max-inherit-rr 0 version 1 migrate offset 0 migrate hash 0x0 pool
LustreError: 26433:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[0] [0x280000400:0x3a:0x0]
LustreError: 26433:0:(lustre_lmv.h:142:lsm_md_dump()) magic 0xcd20cd0 stripe count 2 master mdt 2 hash type crush:0x82000003 max-inherit 0 max-inherit-rr 0 version 0 migrate offset 1 migrate hash 0x2 pool
LustreError: 26433:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[0] [0x280000400:0x3a:0x0]
LustreError: 26433:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[1] [0x240000404:0xdcb:0x0]
LustreError: 26435:0:(lustre_lmv.h:149:lsm_md_dump()) stripe[1] [0x240000404:0xdcb:0x0]
Lustre: dir [0x280000405:0x1c6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 13 previous similar messages
Lustre: 9948:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000408:0x5e:0x0] with magic=0xbd60bd0
Lustre: 9948:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 11 previous similar messages
LustreError: 31980:0:(out_handler.c:910:out_tx_end()) lustre-MDT0000-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
LustreError: 1495:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 1495:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
Lustre: 9437:0:(lod_lov.c:1315:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000408:0xec:0x0] with magic=0xbd60bd0
Lustre: 9437:0:(lod_lov.c:1315:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 11715:0:(llog_osd.c:626:llog_osd_write_rec()) lustre-MDT0000-osp-MDT0002: index 1523 already set in llog bitmap [0x200000402:0x2:0x0]