LustreError: 28453:0:(llog_osd.c:626:llog_osd_write_rec()) LBUG
Pid: 28453, comm: mdt05_021 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] libcfs_call_trace+0x90/0xf0 [libcfs]
[<0>] lbug_with_loc+0x4c/0xa0 [libcfs]
[<0>] llog_osd_write_rec+0x172c/0x1ba0 [obdclass]
[<0>] llog_write_rec+0x290/0x590 [obdclass]
[<0>] llog_cat_add_rec+0x201/0xa10 [obdclass]
[<0>] llog_add+0x17f/0x1f0 [obdclass]
[<0>] sub_updates_write+0x303/0xe3e [ptlrpc]
[<0>] top_trans_stop+0x49a/0xfb0 [ptlrpc]
[<0>] lod_trans_stop+0x25c/0x340 [lod]
[<0>] mdd_trans_stop+0x28/0x16e [mdd]
[<0>] mdd_migrate_object+0x7fd/0x1120 [mdd]
[<0>] mdd_migrate+0x3e4/0x7f0 [mdd]
[<0>] mdt_reint_migrate+0x1199/0x1c60 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<0>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe

LustreError: 20415:0:(mdt_handler.c:764:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x7:0x0] ACL: rc = -2
Lustre: 10971:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff8801c6e886c0 x1762244179314432/t4294968470(0) o101->b82d144d-1ffd-46d0-be32-46417fd1ae59@0@lo:543/0 lens 376/816 e 0 to 0 dl 1680607138 ref 1 fl Interpret:H/2/0 rc 0/0 job:'cat.0'
LustreError: 30035:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x200000404:0x5b:0x0] mdc close failed: rc = -116
Lustre: 30215:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802f38da0c0 x1762244180340928/t4294969456(0) o101->c4a1fbc9-bd51-4a0b-84bd-593ef500c359@0@lo:597/0 lens 376/840 e 0 to 0 dl 1680607192 ref 1 fl Interpret:H/2/0 rc 0/0 job:'dd.0'
LustreError: 30170:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x200000403:0xde:0x0] mdc close failed: rc = -13
LustreError: 20818:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x122:0x0]: rc = -2
LustreError: 11000:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x122:0x0]: rc = -2
LustreError: 29741:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x280000403:0x122:0x0] mdc close failed: rc = -2
LustreError: 29741:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 32311:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x200000403:0xfe:0x0] mdc close failed: rc = -116
LustreError: 29197:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration.
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000404:0x155:0x0]/0xa): rc = 0
Lustre: 28787:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x2dd:0x0] with magic=0xbd60bd0
17[15119]: segfault at 8 ip 00007f0732d107e8 sp 00007ffc5e2fd580 error 4 in ld-2.17.so[7f0732d05000+22000]
LustreError: 29197:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 3' to finish migration.
LustreError: 21240:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x280000404:0x1cc:0x0] mdc close failed: rc = -2
LustreError: 21240:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 3286:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x210:0x0] with magic=0xbd60bd0
Lustre: 3286:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 20518:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x28d:0x0] with magic=0xbd60bd0
Lustre: 20518:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: 27880:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x284:0x0] with magic=0xbd60bd0
Lustre: 27880:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 26126:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '2' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 2' to finish migration.
Lustre: 29216:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x1f8:0x0] with magic=0xbd60bd0
Lustre: 29216:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 26512:0:(mdt_handler.c:764:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x3bf:0x0] ACL: rc = -2
Lustre: dir [0x280000403:0x3e4:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 21388:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x1f3:0x0] with magic=0xbd60bd0
Lustre: 21388:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 11 previous similar messages
LustreError: 10969:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '12' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 12' to finish migration.
Lustre: dir [0x240000404:0x262:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 32542:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x280000403:0x36a:0x0] mdc close failed: rc = -2
Lustre: 20696:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x317:0x0] with magic=0xbd60bd0
Lustre: 20696:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 3 previous similar messages
9[10097]: segfault at 8 ip 00007fada2ec37e8 sp 00007ffe603a2300 error 4 in ld-2.17.so[7fada2eb8000+22000]
LustreError: 26353:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '3' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 3' to finish migration.
Lustre: 10974:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff88026aba4140 x1762244187211392/t4294973371(0) o101->b82d144d-1ffd-46d0-be32-46417fd1ae59@0@lo:730/0 lens 384/864 e 0 to 0 dl 1680607325 ref 1 fl Interpret:H/2/0 rc 0/0 job:'dd.0'
8[9974]: segfault at 8 ip 00007ffa30ce17e8 sp 00007fff1a5d81f0 error 4 in ld-2.17.so[7ffa30cd6000+22000]
LustreError: 17451:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x240000404:0x30d:0x0] mdc close failed: rc = -2
LustreError: 17451:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x200000404:0x34b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000403:0x373:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
Lustre: dir [0x280000403:0x56e:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000403:0x373:0x0]/0xa): rc = 0
LustreError: 26462:0:(osd_index.c:217:__osd_xattr_load_by_oid()) lustre-MDT0002: can't get bonus, rc = -2
LustreError: 20665:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration.
LustreError: 10976:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '18' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 18' to finish migration.
LustreError: 10976:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 1 previous similar message
1[10324]: segfault at 8 ip 00007f2e1cc367e8 sp 00007ffe0180dec0 error 4 in ld-2.17.so[7f2e1cc2b000+22000]
LustreError: 30260:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '18' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 18' to finish migration.
LustreError: 30260:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 5 previous similar messages
Lustre: dir [0x200000404:0x558:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 18 previous similar messages
Lustre: 8875:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x777:0x0] with magic=0xbd60bd0
Lustre: 8875:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 21 previous similar messages
LustreError: 8305:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '4' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 4' to finish migration.
LustreError: 8305:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 13538:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x240000403:0x9ab:0x0] mdc close failed: rc = -2
LustreError: 13538:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Lustre: dir [0x240000403:0x963:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: ll_ost03_003: service thread pid 16949 was inactive for 62.130 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16949, comm: ll_ost03_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x385/0x750 [ofd]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<0>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 10974:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x778:0x0] with magic=0xbd60bd0
Lustre: 10974:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 7 previous similar messages
LustreError: 10889:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff88028841f840/0x78939ab664e55cfb lrc: 3/0,0 mode: PW/PW res: [0x300000402:0x5e:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000480020020 nid: 0@lo remote: 0x78939ab664e55cf4 expref: 13 pid: 32749 timeout: 89626 lvb_type: 0
LustreError: 11-0: lustre-OST0001-osc-ffff8802849ea548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff8802849ea548: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 5330:0:(osc_request.c:1037:osc_init_grant()) lustre-OST0001-osc-ffff8802849ea548: granted 3407872 but already consumed 5111808
LustreError: 167-0: lustre-OST0001-osc-ffff8802849ea548: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 5344:0:(llite_lib.c:3717:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.32@tcp:/lustre/fid: [0x240000404:0x9d5:0x0]/ may get corrupted (rc -108)
Lustre: 5344:0:(llite_lib.c:3717:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.32@tcp:/lustre/fid: [0x240000403:0xabb:0x0]// may get corrupted (rc -108)
Lustre: 5344:0:(llite_lib.c:3717:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.32@tcp:/lustre/fid: [0x240000403:0xaaa:0x0]// may get corrupted (rc -108)
LustreError: 30407:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-OST0001-osc-ffff8802849ea548: namespace resource [0x300000402:0x5e:0x0].0x0 (ffff88028bebf4c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff8802849ea548: Connection restored to (at 0@lo)
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000403:0x53d:0x0]/0xa): rc = 0
LustreError: 28453:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '15' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 15' to finish migration.
LustreError: 28453:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 5 previous similar messages
LustreError: 17516:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x240000403:0xbb6:0x0] mdc close failed: rc = -116
LustreError: 17516:0:(file.c:242:ll_close_inode_openhandle()) Skipped 3 previous similar messages
Lustre: dir [0x200000403:0x726:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 18 previous similar messages
6[14742]: segfault at 8 ip 00007f3cd64307e8 sp 00007ffd38734ea0 error 4 in ld-2.17.so[7f3cd6425000+22000]
Lustre: 3286:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0xd7e:0x0] with magic=0xbd60bd0
Lustre: 3286:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 29942:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x240000404:0xe6e:0x0] mdc close failed: rc = -13
LustreError: 29942:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Lustre: dir [0x200000403:0x93a:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 6 previous similar messages
3[698]: segfault at 8 ip 00007f41b2e3b7e8 sp 00007ffe76bb8c80 error 4 in ld-2.17.so[7f41b2e30000+22000]
8[7233]: segfault at 8 ip 00007f927e20e7e8 sp 00007ffd3e75e490 error 4 in ld-2.17.so[7f927e203000+22000]
LustreError: 10989:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xe97:0x0]: rc = -2
LustreError: 29191:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '9' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 9' to finish migration.
LustreError: 29191:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 8 previous similar messages
LustreError: 30142:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xe27:0x0]: rc = -2
LustreError: 30142:0:(mdd_object.c:3491:mdd_close()) Skipped 1 previous similar message
Lustre: dir [0x280000403:0xf4e:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
LustreError: 31000:0:(llite_nfs.c:340:ll_dir_get_parent_fid()) lustre: failure inode [0x200000404:0xae6:0x0] get parent: rc = -116
5[10231]: segfault at 8 ip 00007fbd997ab7e8 sp 00007ffe91f92a90 error 4 in ld-2.17.so[7fbd997a0000+22000]
5[12188]: segfault at 8 ip 00007f9357add7e8 sp 00007ffcdd08afe0 error 4 in ld-2.17.so[7f9357ad2000+22000]
Lustre: 10980:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xa9a:0x0] with magic=0xbd60bd0
Lustre: 10980:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 59 previous similar messages
1[14729]: segfault at 8 ip 00007fee111017e8 sp 00007fff803f7320 error 4 in ld-2.17.so[7fee110f6000+22000]
LustreError: 19048:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x280000403:0x12c5:0x0] mdc close failed: rc = -2
LustreError: 19048:0:(file.c:242:ll_close_inode_openhandle()) Skipped 20 previous similar messages
LustreError: 10993:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xea1:0x0]: rc = -2
LustreError: 10993:0:(mdd_object.c:3491:mdd_close()) Skipped 1 previous similar message
LustreError: 10987:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xea1:0x0]: rc = -2
19[26047]: segfault at 8 ip 00007f377ceef7e8 sp 00007ffd9f9f0e70 error 4 in ld-2.17.so[7f377cee4000+22000]
17[30386]: segfault at 8 ip 00007f2e881467e8 sp 00007ffedbd52730 error 4 in ld-2.17.so[7f2e8813b000+22000]
17[2714]: segfault at 8 ip 00007f7e12e297e8 sp 00007ffd6b1c6380 error 4 in ld-2.17.so[7f7e12e1e000+22000]
11[18045]: segfault at 8 ip 00007f0cb45767e8 sp 00007fff6aa93550 error 4 in ld-2.17.so[7f0cb456b000+22000]
Lustre: dir [0x280000404:0x19cb:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 35 previous similar messages
17[26845]: segfault at 8 ip 00007f872496f7e8 sp 00007ffeee3a1e40 error 4 in ld-2.17.so[7f8724964000+22000]
17[27014]: segfault at 8 ip 00007f4450fd87e8 sp 00007ffdceae3160 error 4 in ld-2.17.so[7f4450fcd000+22000]
1[868]: segfault at 8 ip 00007f3a2057d7e8 sp 00007ffd3d7c5930 error 4 in ld-2.17.so[7f3a20572000+22000]
11[9030]: segfault at 8 ip 00007f5df7adb7e8 sp 00007ffe031af2b0 error 4 in ld-2.17.so[7f5df7ad0000+22000]
LustreError: 32394:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '18' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 18' to finish migration.
LustreError: 32394:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 24 previous similar messages
14[4635]: segfault at 0 ip (null) sp 00007ffdb9d0cac8 error 14 in 14[400000+6000]
8[16483]: segfault at 8 ip 00007f20345687e8 sp 00007ffdfcddb930 error 4 in ld-2.17.so[7f203455d000+22000]
LustreError: 13927:0:(out_handler.c:886:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:418: rc = -524
LustreError: 12970:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 12970:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
0[22420]: segfault at 8 ip 00007f45e90d37e8 sp 00007ffc746589d0 error 4 in ld-2.17.so[7f45e90c8000+22000]
13[25521]: segfault at 0 ip (null) sp 00007ffd343aef58 error 14 in 18[400000+6000]
15[2668]: segfault at 8 ip 00007f6d4b81c7e8 sp 00007ffd7a2609a0 error 4 in ld-2.17.so[7f6d4b811000+22000]
Lustre: 10963:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x23cc:0x0] with magic=0xbd60bd0
Lustre: 10963:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 109 previous similar messages
LustreError: 18629:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802849ea548: inode [0x200000403:0x143f:0x0] mdc close failed: rc = -2
LustreError: 18629:0:(file.c:242:ll_close_inode_openhandle()) Skipped 26 previous similar messages
LustreError: 6899:0:(llite_nfs.c:340:ll_dir_get_parent_fid()) lustre: failure inode [0x200000403:0x1335:0x0] get parent: rc = -116
LustreError: 6899:0:(llite_nfs.c:340:ll_dir_get_parent_fid()) Skipped 3 previous similar messages
10[16163]: segfault at 8 ip 00007f8bfed222fc sp 00007ffec2f00930 error 4 in ld-2.17.so[7f8bfed15000+22000]
LustreError: 28893:0:(mdd_object.c:403:mdd_xattr_get()) lustre-MDD0000: object [0x200000404:0x165c:0x0] not found: rc = -2
Lustre: dir [0x240000403:0x2a53:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 49 previous similar messages
5[1633]: segfault at 8 ip 00007f904e1d07e8 sp 00007ffde713c9e0 error 4 in ld-2.17.so[7f904e1c5000+22000]
5[9356]: segfault at 8 ip 00007f61cc3277e8 sp 00007ffe1b012950 error 4 in ld-2.17.so[7f61cc31c000+22000]
Lustre: mdt07_014: service thread pid 16501 was inactive for 40.002 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16501, comm: mdt07_014 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x52f/0xba0 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0x132/0x2a80 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3e1/0xbe0 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0x8c6/0x1780 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<0>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt04_015: service thread pid 3128 was inactive for 40.055 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 3128, comm: mdt04_015 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0x9fd/0xea0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x1d8/0x520 [mdt]
[<0>] mdt_remote_object_lock+0x2a/0x30 [mdt]
[<0>] mdt_rename_lock+0xbe/0x4e0 [mdt]
[<0>] mdt_reint_migrate+0x7cb/0x1c60 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<0>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 8247, comm: mdt07_015 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x7f7/0xa50 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0x9fd/0xea0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x1d8/0x520 [mdt]
[<0>] mdt_remote_object_lock+0x2a/0x30 [mdt]
[<0>] mdt_rename_lock+0xbe/0x4e0 [mdt]
[<0>] mdt_reint_rename+0x1437/0x29c0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<0>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt03_019: service thread pid 1870 was inactive for 40.011 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_007: service thread pid 28237 was inactive for 84.169 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt01_013: service thread pid 30256 was inactive for 92.103 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt02_009: service thread pid 32394 was inactive for 94.333 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
LustreError: 10889:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff880268e287c0/0x78939ab66556d9da lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x2a53:0x0].0x0 bits 0x12/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x78939ab66556d9cc expref: 887 pid: 10978 timeout: 90968 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff8802849ea548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff8802849ea548: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff8802849ea548: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 1447:0:(llite_lib.c:3322:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 4619:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0001-mdc-ffff8802849ea548: namespace resource [0x240000403:0x2a53:0x0].0x0 (ffff88008987bc40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0001-mdc-ffff8802849ea548: Connection restored to (at 0@lo)
11[5813]: segfault at 8 ip 00007fbe81e847e8 sp 00007ffd22e27c40 error 4 in ld-2.17.so[7fbe81e79000+22000]
LustreError: 28421:0:(mdt_open.c:1235:mdt_cross_open()) lustre-MDT0002: [0x280000404:0x28ce:0x0] doesn't exist!: rc = -14
5[13320]: segfault at 8 ip 00007f23ec53c7e8 sp 00007ffec886b9c0 error 4 in ld-2.17.so[7f23ec531000+22000]
LustreError: 20508:0:(llite_lib.c:1804:ll_update_lsm_md()) lustre: [0x200000403:0x19b1:0x0] dir layout mismatch:
LustreError: 20508:0:(lustre_lmv.h:139:lsm_md_dump()) dump LMV: magic=0xcd20cd0 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 20508:0:(lustre_lmv.h:146:lsm_md_dump()) stripe[0] [0x200000400:0x84:0x0]
LustreError: 20508:0:(lustre_lmv.h:146:lsm_md_dump()) stripe[0] [0x200000400:0x84:0x0]
LustreError: 20508:0:(lustre_lmv.h:146:lsm_md_dump()) stripe[1] [0x240000403:0x2c1c:0x0]
11[4184]: segfault at 8 ip 00007f1a2f1f77e8 sp 00007ffff1a74490 error 4 in ld-2.17.so[7f1a2f1ec000+22000]
LustreError: 30344:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '6' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 6' to finish migration.
LustreError: 30344:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 28 previous similar messages
13[1684]: segfault at 8 ip 00007f5746a8c7e8 sp 00007ffe23f8bea0 error 4 in ld-2.17.so[7f5746a81000+22000]
8[12349]: segfault at 8 ip 00007f6703a137e8 sp 00007fff6cb60c00 error 4 in ld-2.17.so[7f6703a08000+22000]
Lustre: mdt06_006: service thread pid 28421 was inactive for 40.027 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
17[21885]: segfault at 8 ip 00007fbee83277e8 sp 00007ffdc19adeb0 error 4 in ld-2.17.so[7fbee831c000+22000]
5[25886]: segfault at 8 ip 00007f6f103c67e8 sp 00007ffec96e5f50 error 4 in ld-2.17.so[7f6f103bb000+22000]
16[27501]: segfault at 8 ip 00007f6cce1987e8 sp 00007ffdf75b34f0 error 4 in ld-2.17.so[7f6cce18d000+22000]
LustreError: 10965:0:(mdd_object.c:403:mdd_xattr_get()) lustre-MDD0001: object [0x240000403:0x30e4:0x0] not found: rc = -2
17[29819]: segfault at 8 ip 00007f82c2bc67e8 sp 00007fff5a0a5400 error 4 in ld-2.17.so[7f82c2bbb000+22000]
LustreError: 10889:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880251f72d40/0x78939ab66569bc41 lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x18fb:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x78939ab66569bc33 expref: 532 pid: 30232 timeout: 91248 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff880287d90958: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff880287d90958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
LustreError: 167-0: lustre-MDT0000-mdc-ffff880287d90958: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 3932:0:(llite_lib.c:1967:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 2903:0:(file.c:5409:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0x1b2d:0x0] error: rc = -5
LustreError: 32160:0:(llite_lib.c:3322:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 29721:0:(mdc_request.c:1474:mdc_read_page()) lustre-MDT0000-mdc-ffff880287d90958: [0x200000404:0x18fb:0x0] lock enqueue fails: rc = -108
LustreError: 29707:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880287d90958: inode [0x200000404:0x18fb:0x0] mdc close failed: rc = -108
LustreError: 29707:0:(file.c:242:ll_close_inode_openhandle()) Skipped 74 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff880287d90958: Connection restored to (at 0@lo)
LustreError: 29197:0:(ldlm_lockd.c:1475:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880289409bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff88028a25f480/0x78939ab66571c0b4 lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x1bd6:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x78939ab66571c098 expref: 17 pid: 29197 timeout: 0 lvb_type: 0
19[5867]: segfault at 8 ip 00007f4f11ba27e8 sp 00007ffc4200da20 error 4 in ld-2.17.so[7f4f11b97000+22000]
19[4783]: segfault at 8 ip 00007f06409a57e8 sp 00007ffeba51fd70 error 4 in ld-2.17.so[7f064099a000+22000]
LustreError: 30510:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x18fb:0x0]: rc = -2
19[6479]: segfault at 8 ip 00007f257c95b7e8 sp 00007fff26e8b420 error 4 in ld-2.17.so[7f257c950000+22000]
Lustre: 7803:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x2ccc:0x0] with magic=0xbd60bd0
Lustre: 7803:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 83 previous similar messages
19[10764]: segfault at 8 ip 00007f7845bfb7e8 sp 00007ffd7d6e9870 error 4 in ld-2.17.so[7f7845bf0000+22000]
19[14342]: segfault at 8 ip 00007f24564367e8 sp 00007ffec6739e30 error 4 in ld-2.17.so[7f245642b000+22000]
LustreError: 10995:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x30f6:0x0]: rc = -2
LustreError: 10995:0:(mdd_object.c:3491:mdd_close()) Skipped 1 previous similar message
19[24335]: segfault at 8 ip 00007fd69732d7e8 sp 00007ffea22f77f0 error 4 in ld-2.17.so[7fd697322000+22000]
LustreError: 7721:0:(update_trans.c:1084:top_trans_stop()) lustre-MDT0002-osp-MDT0000: stop trans failed: rc = -116
LustreError: 11239:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 11239:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 11239:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 11239:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 11239:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 11239:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 11239:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11239:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 28453:0:(llog_osd.c:624:llog_osd_write_rec()) lustre-MDT0002-osp-MDT0000: index 496 already set in llog bitmap [0x280000401:0x3:0x0]