BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2c6daa067 PUD 2d47ec067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core floppy libata
CPU: 7 PID: 3735 Comm: ll_sa_3493 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802d7b35c40 ti: ffff8802a48c0000 task.ti: ffff8802a48c0000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802a48c3d78 EFLAGS: 00010296
RAX: ffff880278f43090 RBX: 0000000000000008 RCX: 000000010026001a
RDX: 0000000000000026 RSI: ffff880278f433c0 RDI: 0000000000000008
RBP: ffff8802a48c3d88 R08: ffff8802ac042128 R09: 0000000000000000
R10: ffff8802ac0413e8 R11: ffff8802ac042128 R12: 0000000000000000
R13: ffff8800a140a4f0 R14: ffff880278f433c0 R15: ffff8802ac042128
FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000a210a000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1120521>] ll_statahead_thread+0xdd1/0x2020 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa111f750>] ? ll_statahead_handle.constprop.30+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 18198:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/7 is open, migrate only dentry
Lustre: 18198:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 507 < left 922, rollback = 2
Lustre: 18198:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 1/4/0
Lustre: 18198:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 13/922/0
Lustre: 18198:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 18198:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 15/263/4, delete: 3/6/0
Lustre: 18198:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 3/3/0
Lustre: 17163:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e9081940 x1837361153323264/t4294967650(0) o101->b09ae593-5930-4106-9fab-3f290d23c7d5@0@lo:198/0 lens 376/816 e 0 to 0 dl 1752244213 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
chown (18156) used greatest stack depth: 10064 bytes left
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/5 is open, migrate only dentry
Lustre: 11102:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 510 < left 877, rollback = 2
Lustre: 11102:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 11102:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 11102:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11102:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 11102:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11102:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 11102:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11102:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0
Lustre: 11102:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11102:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 11102:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 19775:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/8 is open, migrate only dentry
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 510 < left 788, rollback = 2
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 11090:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0
Lustre: 11090:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 11090:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 12/951/0
Lustre: 11090:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 11090:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/21/0, punch: 0/0/0, quota 7/129/0
Lustre: 11090:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 11090:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 5/141/1, delete: 2/2/0
Lustre: 11090:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 11090:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 1/1/0
Lustre: 11090:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 11082:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x2e:0x0] with magic=0xbd60bd0
Lustre: 17823:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802dd984140 x1837361155213952/t4294968565(0) o101->8cc7706d-621d-4586-b561-4aca88e0b647@0@lo:203/0 lens 376/816 e 0 to 0 dl 1752244218 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/16 is open, migrate only dentry
14[20446]: segfault at 8 ip 00007fda02f837e8 sp 00007fff960f0eb0 error 4 in ld-2.17.so[7fda02f78000+22000]
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12839:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12839:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 12839:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12839:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 12839:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12839:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 12839:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12839:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 12839:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12839:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 11105:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 1990, rollback = 2
Lustre: 11105:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 31 previous similar messages
Lustre: 19374:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 19374:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
LustreError: 11103:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000404:0x8e:0x0]/13 failed: rc = -116
Lustre: 17134:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x38:0x0] with magic=0xbd60bd0
Lustre: 17134:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 18843:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/9, destroy: 0/0/0
Lustre: 18843:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 18843:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 18843:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 18843:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 18843:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 18843:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 12/275/4, delete: 0/0/0
Lustre: 18843:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 18843:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 18843:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 19809:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/12 is open, migrate only dentry
Lustre: 19809:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 3 previous similar messages
LustreError: 11105:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000404:0x87:0x0]/20 failed: rc = -2
Lustre: 23052:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1634, rollback = 2
Lustre: 23052:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 101 previous similar messages
11[23638]: segfault at 8 ip 00007fc99e5b27e8 sp 00007ffc6f618a30 error 4 in ld-2.17.so[7fc99e5a7000+22000]
Lustre: 18198:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 18198:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 3 previous similar messages
LustreError: 20036:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/14 failed: rc = -2
8[24664]: segfault at 8 ip 00007fac06ce87e8 sp 00007ffe189ddb30 error 4 in ld-2.17.so[7fac06cdd000+22000]
Lustre: 18198:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/12, destroy: 0/0/0
Lustre: 18198:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 18198:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 18198:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 18198:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 18198:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 18198:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 11/239/4, delete: 0/0/0
Lustre: 18198:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 18198:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 18198:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 71 previous similar messages
14[24565]: segfault at 8 ip 00007f47d2e267e8 sp 00007ffe2f89e9a0 error 4 in ld-2.17.so[7f47d2e1b000+22000]
LustreError: 19775:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000404:0xe5:0x0]/18 failed: rc = -2
Lustre: dir [0x200000403:0x143:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 18843:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0002: '13' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 13' to finish migration: rc = -1
LustreError: 21008:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0xe5:0x0] migrate mdt count mismatch 2 != 1
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 983, rollback = 2
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 61 previous similar messages
Lustre: 20862:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a3fa3240 x1837361162029184/t4294972255(0) o101->b09ae593-5930-4106-9fab-3f290d23c7d5@0@lo:230/0 lens 376/864 e 0 to 0 dl 1752244245 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
8[27156]: segfault at 8 ip 00007f715b2197e8 sp 00007ffed9fbf300 error 4 in ld-2.17.so[7f715b20e000+22000]
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000404:0xa3:0x0]/7 failed: rc = -2
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 1 previous similar message
0[27396]: segfault at 8 ip 00007ff0b7f3e7e8 sp 00007fff6e88f9f0 error 4 in ld-2.17.so[7ff0b7f33000+22000]
Lustre: 11103:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 11103:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 11103:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 11103:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 11103:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 11103:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 11103:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 12/275/3, delete: 0/0/0
Lustre: 11103:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 11103:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 11103:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 19797:0:(mdt_reint.c:2484:mdt_reint_migrate()) lustre-MDT0002: [0x280000403:0x1:0x0]/2 is open, migrate only dentry
LustreError: 27857:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a66892a8: inode [0x240000403:0x10b:0x0] mdc close failed: rc = -13
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/8 is open, migrate only dentry
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 7 previous similar messages
Lustre: 21279:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 21279:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 23052:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
LustreError: 23052:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x28a:0x0]/4 failed: rc = -9
14[29625]: segfault at 8 ip 00007fe2f717d7e8 sp 00007fff3effa1d0 error 4 in ld-2.17.so[7fe2f7172000+22000]
LustreError: 30045:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x1aa:0x0]: rc = -5
LustreError: 30045:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
4[29850]: segfault at 8 ip 00007f36b931d7e8 sp 00007fff07107740 error 4 in ld-2.17.so[7f36b9312000+22000]
LustreError: 20036:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 3' to finish migration: rc = -1
Lustre: dir [0x280000403:0x16d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 29985:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x2c7:0x0]: rc = -5
LustreError: 29985:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 29985:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 29985:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 30176:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x1c8:0x0] with magic=0xbd60bd0
Lustre: 30176:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 506 < left 727, rollback = 2
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 238 previous similar messages
Lustre: dir [0x200000403:0x277:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 29422:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x200000404:0x240:0x0] : rc = -5
LustreError: 29422:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000404:0x240:0x0] error -5.
Lustre: 14212:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x1ca:0x0] with magic=0xbd60bd0
Lustre: 14212:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 3 previous similar messages
14[31596]: segfault at 8 ip 00007f7b464ed7e8 sp 00007ffe4085b1c0 error 4 in ld-2.17.so[7f7b464e2000+22000]
LustreError: 19670:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 31709:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x2c7:0x0]: rc = -5
LustreError: 31709:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 31709:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 31709:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 18986:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x209:0x0] migrate mdt count mismatch 2 != 1
LustreError: 31751:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x2c7:0x0]: rc = -5
LustreError: 31751:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 18206:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x87:0x0]: rc = -2
LustreError: 31137:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a66892a8: inode [0x240000404:0x87:0x0] mdc close failed: rc = -2
LustreError: 16885:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x87:0x0]: rc = -2
LustreError: 32203:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x240000404:0x87:0x0] mdc close failed: rc = -2
LustreError: 18427:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x1b:0x0]: rc = -2
LustreError: 19797:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0002: '10' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 10' to finish migration: rc = -1
LustreError: 19797:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000403:0x1:0x0]/10 failed: rc = -1
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 5 previous similar messages
Lustre: 19374:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 19374:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
6[1025]: segfault at 8 ip 00007fe1631d97e8 sp 00007ffd81fe72a0 error 4 in ld-2.17.so[7fe1631ce000+22000]
Lustre: 30174:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x204:0x0] with magic=0xbd60bd0
Lustre: 30174:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: 21310:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0
Lustre: 21310:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 367 previous similar messages
Lustre: 21310:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 21310:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 367 previous similar messages
Lustre: 21310:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 21310:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 367 previous similar messages
Lustre: 21310:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 11/239/2, delete: 0/0/0
Lustre: 21310:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 367 previous similar messages
Lustre: 21310:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 21310:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 367 previous similar messages
Lustre: 20036:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/4 is open, migrate only dentry
Lustre: 20036:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 11 previous similar messages
LustreError: 30179:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x169:0x0] migrate mdt count mismatch 1 != 2
Lustre: 11103:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 510 < left 610, rollback = 2
Lustre: 11103:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 264 previous similar messages
LustreError: 6425:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x2c7:0x0]: rc = -5
LustreError: 6425:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 6425:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 6425:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 21287:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 21287:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 18984:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x3e7:0x0] with magic=0xbd60bd0
Lustre: 18984:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 4079:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x32a:0x0]: rc = -2
LustreError: 5163:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x280000403:0x32a:0x0] mdc close failed: rc = -2
LustreError: 5163:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 19809:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 17' to finish migration: rc = -1
18[8104]: segfault at 1c ip 00000000004017b0 sp 00007ffeb1ffff30 error 6 in 18[400000+6000]
LustreError: 18843:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000404:0x36e:0x0]/20 failed: rc = -2
LustreError: 18843:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 7 previous similar messages
LustreError: 9134:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x3de:0x0]: rc = -5
LustreError: 9134:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 9134:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 9134:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 4 previous similar messages
18[9216]: segfault at 406000 ip 0000000000406000 sp 00007ffd9c838018 error 14 in 18[606000+1000]
LustreError: 8432:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x240000403:0x403:0x0] dir layout mismatch:
LustreError: 8432:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 8432:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x1c:0x0]
LustreError: 8432:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
Lustre: 12837:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12837:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 13 previous similar messages
traps: 11[10105] trap invalid opcode ip:4059fe sp:7ffdc3f61398 error:0 in 11[400000+6000]
LustreError: 18982:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x231:0x0] migrate mdt count mismatch 1 != 2
LustreError: 9106:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0001: '9' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 9' to finish migration: rc = -1
LustreError: 11086:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xa3:0x0]: rc = -2
LustreError: 9959:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x240000404:0xa3:0x0] mdc close failed: rc = -2
LustreError: 20036:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0001: '16' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 16' to finish migration: rc = -1
LustreError: 18186:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x562:0x0]: rc = -5
LustreError: 18186:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 18186:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 18186:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 11103:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 1/4/0
Lustre: 11103:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 743 previous similar messages
Lustre: 11103:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 17/1203/0
Lustre: 11103:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 743 previous similar messages
Lustre: 11103:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 8/130/0
Lustre: 11103:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 743 previous similar messages
Lustre: 11103:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 16/299/3, delete: 3/6/0
Lustre: 11103:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 743 previous similar messages
Lustre: 11103:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 9/9/0, ref_del: 3/3/0
Lustre: 11103:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 743 previous similar messages
LustreError: 12408:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x240000403:0x562:0x0] : rc = -5
LustreError: 12408:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000403:0x562:0x0] error -5.
LustreError: 19768:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x240000403:0x562:0x0] : rc = -5
Lustre: 17008:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x53c:0x0] with magic=0xbd60bd0
Lustre: 17008:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 13 previous similar messages
traps: 5[15592] trap invalid opcode ip:405955 sp:969b1469 error:0 in 5[400000+6000]
1[22527]: segfault at 8 ip 00007fdb697a97e8 sp 00007fff02e00a00 error 4 in ld-2.17.so[7fdb6979e000+22000]
Lustre: 18300:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18300:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 21 previous similar messages
9[29447]: segfault at 8 ip 00007fed1c1d17e8 sp 00007ffec51a26d0 error 4 in ld-2.17.so[7fed1c1c6000+22000]
LustreError: 15592:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x200000403:0x63f:0x0] mdc close failed: rc = -13
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/2 is open, migrate only dentry
Lustre: 11103:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 12 previous similar messages
Lustre: 11103:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 501 < left 699, rollback = 2
Lustre: 11103:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 710 previous similar messages
9[2309]: segfault at 8 ip 00007f993ab8b7e8 sp 00007ffd7db74690 error 4 in ld-2.17.so[7f993ab80000+22000]
Lustre: mdt_io00_011: service thread pid 21321 was inactive for 40.130 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 21321, comm: mdt_io00_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0x57/0xf0 [mdt]
[<0>] mdt_reint_rename+0xde7/0x3ae0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 11093, comm: mdt_out00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1854/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 30205, comm: mdt00_031 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_out00_000: service thread pid 11090 was inactive for 40.078 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_016: service thread pid 17927 was inactive for 40.116 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
10[6325]: segfault at 8 ip 00007fd9a2c567e8 sp 00007fff95ac7870 error 4 in ld-2.17.so[7fd9a2c4b000+22000]
Lustre: mdt00_012: service thread pid 17804 was inactive for 72.066 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 10797:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88009d809a80/0xe0176a8f89dcf357 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x4dd:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe0176a8f89dcf166 expref: 401 pid: 11093 timeout: 355 lvb_type: 0
Lustre: mdt00_029: service thread pid 30176 completed after 84.831s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 17633 completed after 85.141s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 12863 completed after 85.131s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_004: service thread pid 29429 completed after 99.450s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_032: service thread pid 6029 completed after 85.185s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_026: service thread pid 25831 completed after 85.189s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_011: service thread pid 21321 completed after 100.339s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_000: service thread pid 11090 completed after 99.903s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 30205:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e55cb7e8 ns: mdt-lustre-MDT0001_UUID lock: ffff8800b3c96580/0xe0176a8f89dcf778 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x5a0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xe0176a8f89dcf75c expref: 21 pid: 30205 timeout: 0 lvb_type: 0
LustreError: lustre-MDT0001-mdc-ffff8800a66892a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: mdt00_005: service thread pid 17008 completed after 85.023s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0001-mdc-ffff8800a66892a8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff8800a66892a8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
Lustre: mdt00_012: service thread pid 17804 completed after 85.180s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_031: service thread pid 30205 completed after 100.308s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 17134 completed after 85.033s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_025: service thread pid 22519 completed after 85.032s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 17122 completed after 85.055s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_001: service thread pid 11093 completed after 100.346s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_024: service thread pid 21008 completed after 85.057s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 20275:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x1:0x0] error: rc = -5
LustreError: 20275:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Lustre: mdt00_009: service thread pid 17163 completed after 85.005s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 11083 completed after 85.194s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_033: service thread pid 7043 completed after 85.019s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 17830 completed after 85.244s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 2290:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: mdt00_016: service thread pid 17927 completed after 99.335s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 28598:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0001-mdc-ffff8800a66892a8: [0x240000402:0x15:0x0] lock enqueue fails: rc = -108
Lustre: dir [0x200000403:0xa60:0x0] stripe 1 readdir failed: -108, directory is partially accessed!
LustreError: 28978:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a66892a8: inode [0x240000403:0x3f1:0x0] mdc close failed: rc = -108
LustreError: 28978:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: lustre-MDT0001-mdc-ffff8800a66892a8: Connection restored to 192.168.123.107@tcp (at 0@lo)
LustreError: 17615:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x5d4:0x0]: rc = -2
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000404:0x392:0x0]/8 failed: rc = -2
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 8 previous similar messages
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 618, rollback = 7
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 12839:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12839:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 181 previous similar messages
Lustre: 12839:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 12839:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 181 previous similar messages
Lustre: 12839:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12839:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 181 previous similar messages
Lustre: 12839:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12839:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 181 previous similar messages
Lustre: 12839:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12839:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 181 previous similar messages
19[26304]: segfault at 8 ip 00007fb91781f7e8 sp 00007fff89feba40 error 4 in ld-2.17.so[7fb917814000+22000]
LustreError: 4079:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xaf0:0x0]: rc = -2
LustreError: 26364:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a66892a8: inode [0x200000403:0xa60:0x0] mdc close failed: rc = -2
LustreError: 26364:0:(file.c:248:ll_close_inode_openhandle()) Skipped 12 previous similar messages
LustreError: 23052:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 17' to finish migration: rc = -1
LustreError: 30914:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x199f:0x0]: rc = -5
LustreError: 30914:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 10 previous similar messages
LustreError: 30914:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 30914:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 10 previous similar messages
4[30335]: segfault at 8 ip 00007f52701857e8 sp 00007ffc62d75ec0 error 4 in ld-2.17.so[7f527017a000+22000]
Lustre: 19670:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 504 < left 1606, rollback = 2
Lustre: 19670:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 388 previous similar messages
Lustre: 21310:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/14 is open, migrate only dentry
Lustre: 21310:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 17 previous similar messages
Lustre: dir [0x240000405:0x11c:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
14[4096]: segfault at 8 ip 00007f27c2ce77e8 sp 00007fff8c18c770 error 4 in ld-2.17.so[7f27c2cdc000+22000]
8[8046]: segfault at 8 ip 00007fedcb2ba7e8 sp 00007fffe5a9d340 error 4 in ld-2.17.so[7fedcb2af000+22000]
LustreError: 14014:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x986:0x0]: rc = -5
LustreError: 14014:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 11 previous similar messages
LustreError: 17270:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 17270:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 12 previous similar messages
Lustre: 6029:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0xae7:0x0] with magic=0xbd60bd0
Lustre: 6029:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 11 previous similar messages
LustreError: 18659:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x240000403:0x842:0x0] mdc close failed: rc = -13
3[24458]: segfault at 8 ip 00007f22526e17e8 sp 00007ffe539a28a0 error 4 in ld-2.17.so[7f22526d6000+22000]
Lustre: 12837:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12837:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 109 previous similar messages
19[27517]: segfault at 8 ip 00007ff1e6ce17e8 sp 00007fffd0160e70 error 4 in ld-2.17.so[7ff1e6cd6000+22000]
18[27819]: segfault at 8 ip 00007f12f9aa47e8 sp 00007fffa676f300 error 4 in ld-2.17.so[7f12f9a99000+22000]
0[27974]: segfault at 0 ip 0000000000401d60 sp 00007ffc4b5cbd38 error 6 in 0[400000+6000]
13[28479]: segfault at 8 ip 00007fc111ca07e8 sp 00007ffd9f18da60 error 4 in ld-2.17.so[7fc111c95000+22000]
14[29010]: segfault at 8 ip 00007f3bd1c207e8 sp 00007ffcc4a49540 error 4 in ld-2.17.so[7f3bd1c15000+22000]
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/18 failed: rc = -2
LustreError: 19797:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 11 previous similar messages
LustreError: 21967:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x1d53:0x0]: rc = -2
LustreError: 21967:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 19797:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0001: '3' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 3' to finish migration: rc = -1
LustreError: 19797:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
Lustre: dir [0x200000404:0x10a6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
5[5011]: segfault at 8 ip 00007fe6b653d7e8 sp 00007ffc8abbd880 error 4 in ld-2.17.so[7fe6b6532000+22000]
LustreError: 10797:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802e1ff7c00/0xe0176a8f8a087c1b lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x1209:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe0176a8f8a087b73 expref: 952 pid: 7760 timeout: 634 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8800a668ae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a668ae98: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8800a668ae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 4559:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x200000404:0x1079:0x0] mdc close failed: rc = -108
LustreError: 4559:0:(file.c:248:ll_close_inode_openhandle()) Skipped 3 previous similar messages
LustreError: 6303:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0x1209:0x0] error: rc = -5
LustreError: 6303:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 120 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a668ae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
Lustre: 17162:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0xad2:0x0] with magic=0xbd60bd0
Lustre: 17162:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 18300:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18300:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 825 previous similar messages
Lustre: 18300:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18300:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 825 previous similar messages
Lustre: 18300:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 18300:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 825 previous similar messages
Lustre: 18300:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18300:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 825 previous similar messages
Lustre: 18300:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18300:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 825 previous similar messages
LustreError: 6600:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xafa:0x0]: rc = -5
LustreError: 6600:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 25 previous similar messages
LustreError: 6600:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 6600:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 24 previous similar messages
4[20439]: segfault at 0 ip 0000000000403e5f sp 00007ffef5b1d040 error 6 in 4[400000+6000]
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 922, rollback = 2
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 590 previous similar messages
LustreError: 924:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 14 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 12264:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x240000403:0xb3d:0x0] get parent: rc = -116
Lustre: 19797:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/15 is open, migrate only dentry
Lustre: 19797:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 26 previous similar messages
18[4766]: segfault at 8 ip 00007f8c4d84c7e8 sp 00007ffc5b964420 error 4 in ld-2.17.so[7f8c4d841000+22000]
0[5231]: segfault at 8 ip 00007f9cd2cca7e8 sp 00007ffe7a2d95e0 error 4 in ld-2.17.so[7f9cd2cbf000+22000]
12[9544]: segfault at 8 ip 00007f47addca7e8 sp 00007ffcfae802c0 error 4 in ld-2.17.so[7f47addbf000+22000]
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 582, rollback = 7
Lustre: 12839:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 87 previous similar messages
13[23431]: segfault at 0 ip (null) sp 00007ffff971e9e8 error 14 in 13[400000+6000]
12[23501]: segfault at 0 ip (null) sp 00007ffc67b51358 error 14 in 12[400000+6000]
18[27059]: segfault at 8 ip 00007fa3fa0967e8 sp 00007fff8a805570 error 4 in ld-2.17.so[7fa3fa08b000+22000]
Lustre: 18066:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x2459:0x0] with magic=0xbd60bd0
Lustre: 18066:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 7 previous similar messages
11[7200]: segfault at 8 ip 00007f74da78b7e8 sp 00007ffc7bd53880 error 4 in ld-2.17.so[7f74da780000+22000]
18[7446]: segfault at 8 ip 00007f7a9637f7e8 sp 00007ffcda8e69b0 error 4 in ld-2.17.so[7f7a96374000+22000]
4[8099]: segfault at 8 ip 00007f6d95e9c7e8 sp 00007ffd82f6c070 error 4 in ld-2.17.so[7f6d95e91000+22000]
cp (7976) used greatest stack depth: 9920 bytes left
4[8227]: segfault at 8 ip 00007f16971237e8 sp 00007ffcf58288d0 error 4 in ld-2.17.so[7f1697118000+22000]
14[9220]: segfault at 8 ip 00007f59a389c7e8 sp 00007ffe8c9408a0 error 4 in ld-2.17.so[7f59a3891000+22000]
LustreError: 14888:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/4 failed: rc = -2
LustreError: 14888:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 11 previous similar messages
10[26203]: segfault at 1c ip 00000000004017b0 sp 00007ffc2ac0b1d0 error 6 in 10[400000+6000]
10[26424]: segfault at 8 ip 00007f1fa034f7e8 sp 00007ffc6d84b650 error 4 in ld-2.17.so[7f1fa0344000+22000]
LustreError: 18206:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x1241:0x0]: rc = -2
LustreError: 18206:0:(mdd_object.c:3901:mdd_close()) Skipped 2 previous similar messages
LustreError: 13199:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a66892a8: inode [0x240000403:0x1241:0x0] mdc close failed: rc = -2
LustreError: 13199:0:(file.c:248:ll_close_inode_openhandle()) Skipped 36 previous similar messages
traps: 12[31706] general protection ip:404df6 sp:7ffd7154cd88 error:0 in 12[400000+6000]
5[31971]: segfault at 8 ip 00007f1001cfe7e8 sp 00007ffc72bc1b50 error 4 in ld-2.17.so[7f1001cf3000+22000]
Lustre: 11102:0:(mdt_reint.c:2484:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x1:0x0]/6 is open, migrate only dentry
16[1650]: segfault at 0 ip (null) sp 00007fff2a42b5d8 error 14 in 16[400000+6000]
LustreError: 21310:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0002: '18' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 18' to finish migration: rc = -1
5[3555]: segfault at 8 ip 00007f2d6d0307e8 sp 00007fffcefb6cb0 error 4 in ld-2.17.so[7f2d6d025000+22000]
19[5195]: segfault at 8 ip 00007ff4e905b7e8 sp 00007ffd3e555ad0 error 4 in ld-2.17.so[7ff4e9050000+22000]
8[5272]: segfault at 404592 ip 0000000000404914 sp 00007fffc4e680b8 error 7 in 8[400000+6000]
17[6170]: segfault at 8 ip 00007f27d58d67e8 sp 00007fff152cdd70 error 4 in ld-2.17.so[7f27d58cb000+22000]
Lustre: dir [0x240000405:0xec6:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 18 previous similar messages
LustreError: 8755:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0x115a:0x0]: rc = -5
LustreError: 8755:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 13 previous similar messages
LustreError: 8755:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 8755:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 13 previous similar messages
17[8050]: segfault at 8 ip 00007fc0679eb7e8 sp 00007ffdf4894390 error 4 in ld-2.17.so[7fc0679e0000+22000]
10[12325]: segfault at 8 ip 00007f756c3ed7e8 sp 00007ffdd9729110 error 4 in ld-2.17.so[7f756c3e2000+22000]
LustreError: 22918:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 17163:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0x2a3f:0x0] migrate mdt count mismatch 3 != 2
0[18280]: segfault at 8 ip 00007f9c47d197e8 sp 00007ffea3067700 error 4 in ld-2.17.so[7f9c47d0e000+22000]
5[19318]: segfault at 8 ip 00007ff177b737e8 sp 00007ffea0df38c0 error 4 in ld-2.17.so[7ff177b68000+22000]
3[20005]: segfault at 8 ip 00007f17c0fce7e8 sp 00007ffe1915b7f0 error 4 in ld-2.17.so[7f17c0fc3000+22000]
Lustre: dir [0x240000405:0x103a:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 15 previous similar messages
LustreError: 4079:0:(mdd_orphans.c:261:mdd_orphan_delete()) lustre-MDD0002: could not delete orphan object [0x280000404:0x2adf:0x0]: rc = -2
LustreError: 4079:0:(mdd_object.c:3955:mdd_close()) lustre-MDD0002: unable to delete [0x280000404:0x2adf:0x0] from orphan list: rc = -2
16[23420]: segfault at 0 ip 0000000000403cf0 sp 00007ffcb17359b8 error 6 in 16[400000+6000]
LustreError: 10797:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802ac65cb40/0xe0176a8f8a5329b5 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1a80:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe0176a8f8a5328ea expref: 900 pid: 17008 timeout: 1084 lvb_type: 0
LustreError: 18984:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e55cc138 ns: mdt-lustre-MDT0002_UUID lock: ffff880299994000/0xe0176a8f8a536f61 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xe0176a8f8a536f53 expref: 356 pid: 18984 timeout: 0 lvb_type: 0
LustreError: 18984:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: lustre-MDT0002-mdc-ffff8800a66892a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a66892a8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8800a66892a8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: Skipped 4 previous similar messages
LustreError: 26324:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 26324:0:(llite_lib.c:2023:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 26734:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -5
LustreError: 26734:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 4 previous similar messages
LustreError: 23528:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0002-mdc-ffff8800a66892a8: [0x280000400:0x58:0x0] lock enqueue fails: rc = -108
Lustre: dir [0x280000404:0x2a3f:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0002-mdc-ffff8800a66892a8: Connection restored to 192.168.123.107@tcp (at 0@lo)
6[4114]: segfault at 1c ip 00000000004017b0 sp 00007ffda04b1890 error 6 in 6[400000+6000]
LustreError: 20036:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
10[5703]: segfault at 8 ip 00007f66578e17e8 sp 00007ffd48df1580 error 4 in ld-2.17.so[7f66578d6000+22000]
LustreError: 14888:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0000: '7' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 7' to finish migration: rc = -1
LustreError: 14888:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) Skipped 9 previous similar messages
LustreError: 18986:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x79:0x0] migrate mdt count mismatch 3 != 2
LustreError: 18206:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000405:0xfff:0x0]: rc = -2
LustreError: 18206:0:(mdd_object.c:3901:mdd_close()) Skipped 4 previous similar messages
Lustre: 7043:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x1561:0x0] with magic=0xbd60bd0
Lustre: 7043:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 37 previous similar messages
5[8372]: segfault at 8 ip 00007faa79a327e8 sp 00007fffaab9d7f0 error 4 in ld-2.17.so[7faa79a27000+22000]
9[8437]: segfault at 8 ip 00007fc4a39c67e8 sp 00007ffdf7d96110 error 4 in ld-2.17.so[7fc4a39bb000+22000]
LustreError: 13443:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x280000403:0x1cf0:0x0] : rc = -5
LustreError: 13443:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000403:0x1cf0:0x0] error -5.
LustreError: 11878:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x280000405:0x207:0x0] : rc = -5
LustreError: 11878:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000405:0x207:0x0] error -5.
Lustre: 21284:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 21284:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 2967 previous similar messages
Lustre: 21284:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 21284:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 2967 previous similar messages
Lustre: 21284:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 21284:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 2967 previous similar messages
Lustre: 21284:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 21284:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 2967 previous similar messages
Lustre: 21284:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21284:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 2967 previous similar messages
LustreError: 21318:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x280000405:0x207:0x0] : rc = -5
LustreError: 22689:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x200000405:0x16e2:0x0] : rc = -5
LustreError: 22689:0:(lov_object.c:1350:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 22689:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000405:0x16e2:0x0] error -5.
LustreError: 22720:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x240000403:0x1aff:0x0] : rc = -5
LustreError: 22720:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000403:0x1aff:0x0] error -5.
LustreError: 18262:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 17830:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x1fbe:0x0] migrate mdt count mismatch 2 != 1
4[31088]: segfault at 8 ip 00007f996f22f7e8 sp 00007fff54df40c0 error 4 in ld-2.17.so[7f996f224000+22000]
Lustre: 18262:0:(mdt_reint.c:2484:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x1:0x0]/10 is open, migrate only dentry
ptlrpc_watchdog_fire: 18 callbacks suppressed
Lustre: mdt00_033: service thread pid 7043 was inactive for 40.103 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 7043, comm: mdt00_033 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xc5a/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
Lustre: mdt00_000: service thread pid 11081 was inactive for 40.063 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 8 previous similar messages
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 17830, comm: mdt00_014 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xc5a/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 17932, comm: mdt00_017 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xc5a/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 10797:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802a4b49a80/0xe0176a8f8a6fcc05 lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x2f1f:0x0].0x0 bits 0x19/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe0176a8f8a6fcbdb expref: 1248 pid: 14212 timeout: 1293 lvb_type: 0
Lustre: mdt00_011: service thread pid 17758 completed after 99.174s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 18984 completed after 99.220s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 17932 completed after 99.244s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 11081 completed after 99.219s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_027: service thread pid 30158 completed after 99.247s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 17830 completed after 99.222s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a66892a8: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800a66892a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_035: service thread pid 27923 completed after 99.252s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 11822:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752245394 with bad export cookie 16147492153683927162
Lustre: mdt00_033: service thread pid 7043 completed after 99.289s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a66892a8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 31603:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000404:0x2f1f:0x0] error -108.
LustreError: 31798:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 31798:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 21 previous similar messages
LustreError: 31877:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0000-mdc-ffff8800a66892a8: [0x200000403:0x1:0x0] lock enqueue fails: rc = -108
LustreError: 31877:0:(mdc_request.c:1470:mdc_read_page()) Skipped 6 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a66892a8: Connection restored to 192.168.123.107@tcp (at 0@lo)
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 503 < left 2168, rollback = 2
Lustre: 19797:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3074 previous similar messages
Lustre: 18262:0:(mdd_dir.c:4838:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/16 is open, migrate only dentry
Lustre: 18262:0:(mdd_dir.c:4838:mdd_migrate_object()) Skipped 91 previous similar messages
3[1795]: segfault at 8 ip 00007f8ab02c97e8 sp 00007ffd6d194910 error 4 in ld-2.17.so[7f8ab02be000+22000]
Lustre: 32728:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32728:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 291 previous similar messages
10[3273]: segfault at 8 ip 00007fa8876717e8 sp 00007ffc49690f00 error 4 in ld-2.17.so[7fa887666000+22000]
LustreError: 12863:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x615:0x0] migrate mdt count mismatch 1 != 2
2[3259]: segfault at 8 ip 00007efc0ea717e8 sp 00007ffd8d392910 error 4 in ld-2.17.so[7efc0ea66000+22000]
LustreError: 17758:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0xf5:0x0] migrate mdt count mismatch 2 != 1
LustreError: 11082:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1be3:0x0] migrate mdt count mismatch 2 != 3
LustreError: 12088:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x200000405:0x1dcb:0x0] : rc = -5
LustreError: 12088:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000405:0x1dcb:0x0] error -5.
LustreError: 19809:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) lustre-MDD0002: '1' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 1' to finish migration: rc = -1
LustreError: 19809:0:(mdd_dir.c:4759:mdd_migrate_cmd_check()) Skipped 20 previous similar messages
19[19099]: segfault at 8 ip 00007ffb753ae7e8 sp 00007ffd449ca640 error 4 in ld-2.17.so[7ffb753a3000+22000]
19[19356]: segfault at 8 ip 00007f642e0b37e8 sp 00007fff9f5486e0 error 4 in ld-2.17.so[7f642e0a8000+22000]
9[20994]: segfault at 8 ip 00007f0d3a0437e8 sp 00007ffefbf1afe0 error 4 in ld-2.17.so[7f0d3a038000+22000]
LustreError: 25261:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x280000405:0x943:0x0] : rc = -5
LustreError: 25261:0:(lov_object.c:1350:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 25261:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000405:0x943:0x0] error -5.
LustreError: 25261:0:(vvp_io.c:1909:vvp_io_init()) Skipped 1 previous similar message
LustreError: 11086:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x20be:0x0]: rc = -2
LustreError: 11086:0:(mdd_object.c:3901:mdd_close()) Skipped 13 previous similar messages
LustreError: 19809:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x280000403:0x253c:0x0]/19 failed: rc = -2
LustreError: 19809:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 92 previous similar messages
16[30710]: segfault at 0 ip (null) sp 00007ffc811cd228 error 14 in 16[400000+6000]
4[30681]: segfault at 8 ip 00007f488d9837e8 sp 00007ffed5de50e0 error 4 in ld-2.17.so[7f488d978000+22000]
LustreError: 30560:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a668ae98: inode [0x280000405:0x9dd:0x0] mdc close failed: rc = -2
LustreError: 30560:0:(file.c:248:ll_close_inode_openhandle()) Skipped 77 previous similar messages
LustreError: 18066:0:(mdt_xattr.c:406:mdt_dir_layout_update()) lustre-MDT0000: [0x200000406:0x62c:0x0] migrate mdt count mismatch 2 != 3
LustreError: 31480:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a66892a8: cannot apply new layout on [0x240000403:0x2278:0x0] : rc = -5
LustreError: 31480:0:(lov_object.c:1350:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 31480:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000403:0x2278:0x0] error -5.
LustreError: 31480:0:(vvp_io.c:1909:vvp_io_init()) Skipped 1 previous similar message
Lustre: dir [0x200000406:0x744:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 15 previous similar messages
LustreError: 7555:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000405:0x95a:0x0]: rc = -5
LustreError: 7555:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 190 previous similar messages
LustreError: 7555:0:(llite_lib.c:3770:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 7555:0:(llite_lib.c:3770:ll_prep_inode()) Skipped 190 previous similar messages
10[7620]: segfault at 8 ip 00007f01e3f347e8 sp 00007fff0adf6a10 error 4 in ld-2.17.so[7f01e3f29000+22000]
LustreError: 16933:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x280000405:0x943:0x0] : rc = -5
LustreError: 16933:0:(lov_object.c:1350:lov_layout_change()) Skipped 2 previous similar messages
16[17395]: segfault at 8 ip 00007fd8c007e7e8 sp 00007fff320994c0 error 4 in ld-2.17.so[7fd8c0073000+22000]
10[17477]: segfault at 0 ip (null) sp 00007fff9f3fc2c8 error 14 in 10[400000+6000]
LustreError: 12268:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000405:0x1f28:0x0] error -5.
LustreError: 158:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 158:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 174:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 10797:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802abc39e40/0xe0176a8f8aa4d8d7 lrc: 3/0,0 mode: CR/CR res: [0x280000403:0x2a7d:0x0].0x0 bits 0xa/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe0176a8f8aa4d87c expref: 727 pid: 11082 timeout: 1626 lvb_type: 0
LustreError: lustre-MDT0002-mdc-ffff8800a668ae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a668ae98: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8800a668ae98: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 26853:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 26814:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x29cd:0x0] error: rc = -108
LustreError: 26814:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 15 previous similar messages
Lustre: lustre-MDT0002-mdc-ffff8800a668ae98: Connection restored to 192.168.123.107@tcp (at 0@lo)
Lustre: 17252:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000406:0xd74:0x0] with magic=0xbd60bd0
Lustre: 17252:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 165 previous similar messages
LustreError: 30399:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8800a668ae98: cannot apply new layout on [0x280000405:0x943:0x0] : rc = -5
LustreError: 30399:0:(lov_object.c:1350:lov_layout_change()) Skipped 16 previous similar messages
LustreError: 406:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 5 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
13[31518]: segfault at 8 ip 00007fbae7bed7e8 sp 00007ffe95976680 error 4 in ld-2.17.so[7fbae7be2000+22000]
13[9123]: segfault at 8 ip 00007fc743a067e8 sp 00007fffd1642810 error 4 in ld-2.17.so[7fc7439fb000+22000]
19[25134]: segfault at 8 ip 00007f5c96e387e8 sp 00007ffe51e1a100 error 4 in ld-2.17.so[7f5c96e2d000+22000]
11[24772]: segfault at 0 ip (null) sp 00007ffdc3da5be8 error 14 in 11[400000+6000]
9[31033]: segfault at 8 ip 00007f5e49a087e8 sp 00007fff07f71d90 error 4 in ld-2.17.so[7f5e499fd000+22000]
11[31862]: segfault at 8 ip 00007efca45137e8 sp 00007ffca5b4a2c0 error 4 in ld-2.17.so[7efca4508000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2bb379067 PUD 2984c4067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata
CPU: 5 PID: 8983 Comm: ll_sa_8797 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b8e3760 ti: ffff8802b480c000 task.ti: ffff8802b480c000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802b480fd78 EFLAGS: 00010296
RAX: ffff88008f614d10 RBX: 0000000000000008 RCX: 0000000100260024
RDX: 0000000000000026 RSI: ffff88008f615040 RDI: 0000000000000008
RBP: ffff8802b480fd88 R08: ffff8802ed703ef8 R09: 0000000000000000
R10: ffff8802ed7009f8 R11: ffff8802ed703ef8 R12: 0000000000000000
R13: ffff88031b8e0010 R14: ffff88008f615040 R15: ffff8802ed703ef8
FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002d8a0e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1127791>] ll_statahead_thread+0xdd1/0x2020 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa11269c0>] ? ll_statahead_handle.constprop.30+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 15457:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a8576940 x1837073806699648/t4294968442(0) o101->6abfd204-2290-4b04-b77a-72a76796ca30@0@lo:219/0 lens 376/864 e 0 to 0 dl 1751970169 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 558, rollback = 7
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/558/0, punch: 0/0/0, quota 1/3/0
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15457:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x28b:0x0] with magic=0xbd60bd0
11[24401]: segfault at 0 ip (null) sp 00007ffe3bda1058 error 14 in 11[400000+6000]
5[25432]: segfault at 8 ip 00007f3b4b3ad7e8 sp 00007fffcb536e90 error 4 in ld-2.17.so[7f3b4b3a2000+22000]
Lustre: 15118:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15118:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15118:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15118:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 15118:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15118:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15118:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15118:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15118:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 15118:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15118:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15118:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15118:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15118:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 15118:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15118:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15118:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15118:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15118:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10478:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10478:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 10478:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10478:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
2[29021]: segfault at 8 ip 00007f9ab52867e8 sp 00007fff1c619ff0 error 4 in ld-2.17.so[7f9ab527b000+22000]
0[31981]: segfault at 8 ip 00007f5dc0b677e8 sp 00007ffce809fd30 error 4 in ld-2.17.so[7f5dc0b5c000+22000]
Lustre: 10478:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 618, rollback = 7
Lustre: 10478:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 10478:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10478:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 7 previous similar messages
11[2329]: segfault at 8 ip 00007f06c9c947e8 sp 00007ffca119a6c0 error 4 in ld-2.17.so[7f06c9c89000+22000]
Lustre: 16885:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x6bf:0x0] with magic=0xbd60bd0
Lustre: 16885:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9645:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x710:0x0] with magic=0xbd60bd0
Lustre: 9645:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
0[6265]: segfault at 8 ip 00007fc4e8aa57e8 sp 00007fff239ec220 error 4 in ld-2.17.so[7fc4e8a9a000+22000]
7[6855]: segfault at 8 ip 00007faaac3ae7e8 sp 00007ffddcd3c7d0 error 4 in ld-2.17.so[7faaac3a3000+22000]
16[9886]: segfault at 8 ip 00007f87aa6b57e8 sp 00007ffd73be4470 error 4 in ld-2.17.so[7f87aa6aa000+22000]
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 21 previous similar messages
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 15625:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xbbc:0x0] with magic=0xbd60bd0
Lustre: 15625:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
9[23884]: segfault at 8 ip 00007f9ac61107e8 sp 00007ffc2b0c6ef0 error 4 in ld-2.17.so[7f9ac6105000+22000]
Lustre: 15625:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0xe17:0x0] with magic=0xbd60bd0
Lustre: 15625:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 89 previous similar messages
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10478:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 91 previous similar messages
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10478:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 91 previous similar messages
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10478:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 91 previous similar messages
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10478:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 91 previous similar messages
Lustre: 15457:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0xf78:0x0] with magic=0xbd60bd0
Lustre: 15457:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
7[31699]: segfault at 8 ip 00007fc7172cf7e8 sp 00007fffd23eb7d0 error 4 in ld-2.17.so[7fc7172c4000+22000]
18[987]: segfault at 1c ip 00000000004017b0 sp 00007fff666d0770 error 6 in 18[400000+6000]
Lustre: 9643:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x11af:0x0] with magic=0xbd60bd0
Lustre: 9643:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
17[18571]: segfault at 0 ip 0000000000403e5f sp 00007ffd4a856980 error 6 in 17[400000+6000]
Lustre: 16055:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x16ee:0x0] with magic=0xbd60bd0
Lustre: 16055:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 20564:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000402:0x18cb:0x0] mdc close failed: rc = -13
5[28269]: segfault at 0 ip 0000000000403e5f sp 00007fff68391eb0 error 6 in 2[400000+6000]
1[3790]: segfault at 0 ip 0000000000403e5f sp 00007ffc70ac5680 error 6 in 1[400000+6000]
Lustre: mdt00_005: service thread pid 14983 was inactive for 40.078 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 14983, comm: mdt00_005 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 15332, comm: mdt00_006 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_link+0x8fb/0xff0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 32422, comm: mdt00_014 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_000: service thread pid 9643 was inactive for 40.058 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 9314:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802c5e3b880/0xd06913fc8f3fc9c3 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x2265:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd06913fc8f3fc999 expref: 1044 pid: 31285 timeout: 483 lvb_type: 0
Lustre: mdt00_006: service thread pid 15332 completed after 100.399s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 32422 completed after 100.364s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 14983 completed after 100.387s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 9643:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a9e053d8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802bbd796c0/0xd06913fc8f3fced9 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x2153:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd06913fc8f3fce69 expref: 136 pid: 9643 timeout: 0 lvb_type: 0
Lustre: mdt00_000: service thread pid 9643 completed after 98.447s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802ee839bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802ee839bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802ee839bf8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 13478:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000402:0x2153:0x0] error: rc = -5
LustreError: 13500:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee839bf8: inode [0x200000401:0x1f62:0x0] mdc close failed: rc = -108
LustreError: 13500:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8802ee839bf8: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8802ddfd5940) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff8802ee839bf8: Connection restored to 192.168.123.47@tcp (at 0@lo)
14[13483]: segfault at 0 ip (null) sp 00007ffed8dc5ab8 error 14 in 14[400000+6000]
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10481:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 191 previous similar messages
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10481:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 191 previous similar messages
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10481:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 187 previous similar messages
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10481:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 187 previous similar messages
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10481:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 187 previous similar messages
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10481:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 187 previous similar messages
Lustre: 16885:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x140:0x0] with magic=0xbd60bd0
Lustre: 16885:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
12[21571]: segfault at 8 ip 00007ff3cbb587e8 sp 00007ffd0a0332e0 error 4 in ld-2.17.so[7ff3cbb4d000+22000]
3[21786]: segfault at 8 ip 00007f57755097e8 sp 00007fff53398610 error 4 in ld-2.17.so[7f57754fe000+22000]
12[26795]: segfault at 8 ip 00007f9ea39897e8 sp 00007ffded5f94e0 error 4 in ld-2.17.so[7f9ea397e000+22000]
8[27922]: segfault at 8 ip 00007f37adf467e8 sp 00007ffc92d6d980 error 4 in ld-2.17.so[7f37adf3b000+22000]
7[2214]: segfault at 0 ip (null) sp 00007fff57c64b28 error 14 in 7[400000+6000]
traps: 2[6959] trap invalid opcode ip:405152 sp:7ffe38671968 error:0 in 2[400000+6000]
8[17642]: segfault at 8 ip 00007f3f764b17e8 sp 00007ffcc56a2930 error 4 in ld-2.17.so[7f3f764a6000+22000]
13[17489]: segfault at 8 ip 00007f8ed25567e8 sp 00007ffeced6ded0 error 4 in ld-2.17.so[7f8ed254b000+22000]
6[24778]: segfault at 8 ip 00007f9c43e0f7e8 sp 00007ffc5b8893e0 error 4 in ld-2.17.so[7f9c43e04000+22000]
LustreError: 25875:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000403:0x105c:0x0] mdc close failed: rc = -13
LustreError: 25875:0:(file.c:248:ll_close_inode_openhandle()) Skipped 8 previous similar messages
LustreError: 30869:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000402:0x3370:0x0] mdc close failed: rc = -13
4[10323]: segfault at 8 ip 00007ff9840e17e8 sp 00007ffcf95cb740 error 4 in ld-2.17.so[7ff9840d6000+22000]
LustreError: 10434:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000403:0x16ef:0x0] mdc close failed: rc = -13
LustreError: 15598:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000403:0x18c4:0x0] mdc close failed: rc = -13
4[29340]: segfault at 0 ip 0000000000403e5f sp 00007ffe77cab120 error 6 in 4[400000+6000]
11[32490]: segfault at 8 ip 00007f243be437e8 sp 00007ffe02e739f0 error 4 in ld-2.17.so[7f243be38000+22000]
LustreError: 32766:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000402:0x4071:0x0] mdc close failed: rc = -13
Lustre: mdt_io00_004: service thread pid 5027 was inactive for 40.033 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 9314:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8802d85e8b80/0xd06913fc8f73e69d lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x44a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xd06913fc8f73e688 expref: 15 pid: 10467 timeout: 746 lvb_type: 0
LustreError: lustre-OST0003-osc-ffff8802ee839bf8: operation ost_sync to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff8802ee839bf8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff8802ee839bf8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3740:0:(llite_lib.c:4215:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.47@tcp:/lustre/fid: [0x200000402:0x3c65:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff8802ee839bf8: Connection restored to 192.168.123.47@tcp (at 0@lo)
Lustre: lustre-OST0003-osc-ffff8802ee839bf8: disconnect after 24s idle
LustreError: 9314:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802d6d4d2c0/0xd06913fc8f7ce6d8 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1eec:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd06913fc8f7ce6ae expref: 788 pid: 13028 timeout: 775 lvb_type: 0
LustreError: 15457:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d9c392a8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d622d680/0xd06913fc8f7cf16d lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1eec:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x54a01000000000 nid: 0@lo remote: 0xd06913fc8f7cf151 expref: 598 pid: 15457 timeout: 0 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802ee839bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802ee839bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt_io00_004: service thread pid 5027 completed after 100.836s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802ee839bf8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 15457:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 4 previous similar messages
LustreError: 618:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 618:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
LustreError: 685:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee839bf8: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
LustreError: 331:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1eec:0x0] error -108.
Lustre: lustre-MDT0000-mdc-ffff8802ee839bf8: Connection restored to 192.168.123.47@tcp (at 0@lo)
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 570, rollback = 7
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 331 previous similar messages
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 331 previous similar messages
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 331 previous similar messages
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/570/0, punch: 0/0/0, quota 1/3/0
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 331 previous similar messages
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 331 previous similar messages
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 331 previous similar messages
14[15050]: segfault at 8 ip 00007f3dcfcc87e8 sp 00007ffda0a016d0 error 4 in ld-2.17.so[7f3dcfcbd000+22000]
Lustre: 32422:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x45d7:0x0] with magic=0xbd60bd0
Lustre: 32422:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 13 previous similar messages
11[19461]: segfault at 8 ip 00007f43974067e8 sp 00007ffe4a2e2910 error 4 in ld-2.17.so[7f43973fb000+22000]
16[22157]: segfault at 8 ip 00007f88d89f387a sp 00007ffd555ac640 error 4 in ld-2.17.so[7f88d89e8000+22000]
5[8478]: segfault at 0 ip 0000000000403e5f sp 00007ffc2bbb8680 error 6 in 5[400000+6000]
LustreError: 9827:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000402:0x4f06:0x0] mdc close failed: rc = -13
LustreError: 9827:0:(file.c:248:ll_close_inode_openhandle()) Skipped 9 previous similar messages
Lustre: lustre-OST0000-osc-ffff8802ed4c37e8: disconnect after 22s idle
Lustre: lustre-OST0003-osc-ffff8802ed4c37e8: disconnect after 22s idle
LustreError: 9314:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802c64752c0/0xd06913fc8f9e0ab5 lrc: 3/0,0 mode: CR/CR res: [0x200000404:0x1018:0x0].0x0 bits 0xa/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd06913fc8f9e0a92 expref: 1421 pid: 32422 timeout: 974 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802ed4c37e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8802ed4c37e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 10498:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1751971043 with bad export cookie 15017556407895070725
LustreError: 21291:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ddcbf340 x1837074016313216/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-MDT0000-mdc-ffff8802ed4c37e8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 14634:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000404:0x1018:0x0] error -108.
LustreError: 14526:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0xc38:0x0] error: rc = -108
LustreError: 14526:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 37 previous similar messages
LustreError: 14604:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0000-mdc-ffff8802ed4c37e8: [0x200000401:0x1:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8802ed4c37e8: Connection restored to 192.168.123.47@tcp (at 0@lo)
3[25612]: segfault at 8 ip 00007f63da07d7e8 sp 00007ffdaa0dec90 error 4 in ld-2.17.so[7f63da072000+22000]
1[27758]: segfault at 1c ip 00000000004017b0 sp 00007fff105732f0 error 6 in 1[400000+6000]
18[29132]: segfault at 8 ip 00007f1c88faf7e8 sp 00007ffc16cc4200 error 4 in ld-2.17.so[7f1c88fa4000+22000]
0[29386]: segfault at 8 ip 00007f115ffed7e8 sp 00007ffe783b4b00 error 4 in ld-2.17.so[7f115ffe2000+22000]
0[30659]: segfault at 8 ip 00007f16d1bf07e8 sp 00007ffc4d16c000 error 4 in ld-2.17.so[7f16d1be5000+22000]
9[2340]: segfault at 0 ip 0000000000403e5f sp 00007ffd4cd3f4b0 error 6 in 9[400000+6000]
18[4272]: segfault at 8 ip 00007f4abe85a7e8 sp 00007ffd6a389bd0 error 4 in ld-2.17.so[7f4abe84f000+22000]
9[5311]: segfault at 6fa86d7d ip 0000000000404d03 sp 00007ffd9f081330 error 6 in 9[400000+6000]
10[8055]: segfault at 0 ip 0000000000403e5f sp 00007ffe1c5975c0 error 6 in 10[400000+6000]
LustreError: 13713:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee839bf8: inode [0x200000404:0x1bc6:0x0] mdc close failed: rc = -13
LustreError: 13713:0:(file.c:248:ll_close_inode_openhandle()) Skipped 9 previous similar messages
Lustre: 16055:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x1c9b:0x0] with magic=0xbd60bd0
Lustre: 16055:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 3 previous similar messages
11[17409]: segfault at 8 ip 00007f9f316757e8 sp 00007fffe1718c50 error 4 in ld-2.17.so[7f9f3166a000+22000]
18[22340]: segfault at 0 ip 0000000000403e5f sp 00007fffe6f8aec0 error 6 in 18[400000+6000]
9[25101]: segfault at 8 ip 00007f9f495987e8 sp 00007ffc68aac270 error 4 in ld-2.17.so[7f9f4958d000+22000]
12[26555]: segfault at 0 ip 0000000000403e5f sp 00007ffe12e941b0 error 6 in 12[400000+6000]
19[27758]: segfault at 8 ip 00007f3949ae47e8 sp 00007ffc4cd2ecc0 error 4 in ld-2.17.so[7f3949ad9000+22000]
11[30938]: segfault at 0 ip 0000000000403e5f sp 00007ffe0db8c3a0 error 6 in 11[400000+6000]
7[32233]: segfault at 8 ip 00007f172eb937e8 sp 00007ffe75cfe3f0 error 4 in ld-2.17.so[7f172eb88000+22000]
16[4848]: segfault at 8 ip 00007f7a897c77e8 sp 00007ffc9b0f6410 error 4 in ld-2.17.so[7f7a897bc000+22000]
18[6565]: segfault at 8 ip 00007f0086afa7e8 sp 00007fff83a98430 error 4 in ld-2.17.so[7f0086aef000+22000]
5[10150]: segfault at 8 ip 00007f804f8d87e8 sp 00007ffc77b59300 error 4 in ld-2.17.so[7f804f8cd000+22000]
1[10963]: segfault at 0 ip 0000000000403e5f sp 00007fff25dfb600 error 6 in 1[400000+6000]
8[15542]: segfault at 8 ip 00007faf9a5e57e8 sp 00007fff99c73a40 error 4 in ld-2.17.so[7faf9a5da000+22000]
1[16046]: segfault at 8 ip 00007f33beef07e8 sp 00007fff8d7e8f10 error 4 in ld-2.17.so[7f33beee5000+22000]
14[23423]: segfault at 8 ip 00007fcf971877e8 sp 00007ffc0cdf9ae0 error 4 in ld-2.17.so[7fcf9717c000+22000]
6[24245]: segfault at 8 ip 00007f0bc6d197e8 sp 00007ffc7f552990 error 4 in ld-2.17.so[7f0bc6d0e000+22000]
3[24621]: segfault at 406000 ip 0000000000406000 sp 00007fff370b2238 error 14 in 3[606000+1000]
12[27219]: segfault at 8 ip 00007fa7bc8b37e8 sp 00007fffdd1cbc50 error 4 in ld-2.17.so[7fa7bc8a8000+22000]
6[31943]: segfault at 8 ip 00007f5feed107e8 sp 00007ffc7e895460 error 4 in ld-2.17.so[7f5feed05000+22000]
4[1493]: segfault at 8 ip 00007fed6a43a7e8 sp 00007ffe6f2332e0 error 4 in ld-2.17.so[7fed6a42f000+22000]
7[8047]: segfault at 0 ip 0000000000403e5f sp 00007fff5907e8c0 error 6 in 7[400000+6000]
8[10344]: segfault at 8 ip 00007f082fb917e8 sp 00007ffdeb15e580 error 4 in ld-2.17.so[7f082fb86000+22000]
Lustre: 20833:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 20833:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 579 previous similar messages
Lustre: 20833:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20833:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 20833:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20833:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 20833:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 20833:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 20833:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20833:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 20833:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20833:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 579 previous similar messages
16[15390]: segfault at 8 ip 00007f89f8f4f7e8 sp 00007ffde89c41f0 error 4 in ld-2.17.so[7f89f8f44000+22000]
2[16108]: segfault at 4045c9 ip 0000000000405b29 sp 00007ffc237b77e0 error 7 in 2[400000+6000]
13[16572]: segfault at 8 ip 00007f4e80c327e8 sp 00007fff31edf3d0 error 4 in ld-2.17.so[7f4e80c27000+22000]
9[16906]: segfault at 8 ip 00007fea146167e8 sp 00007fff4fd58c60 error 4 in ld-2.17.so[7fea1460b000+22000]
8[28701]: segfault at 8 ip 00007f18180e77e8 sp 00007ffeb4bd2470 error 4 in ld-2.17.so[7f18180dc000+22000]
1[29927]: segfault at 8 ip 00007fdd779a27e8 sp 00007ffeabedd550 error 4 in ld-2.17.so[7fdd77997000+22000]
3[703]: segfault at 8 ip 00007feff65587e8 sp 00007fff5c2ba7c0 error 4 in ld-2.17.so[7feff654d000+22000]
3[4283]: segfault at 0 ip (null) sp 00007ffdc03046f8 error 14 in 3[400000+6000]
8[5558]: segfault at 8 ip 00007f05e0f347e8 sp 00007ffd0d718e60 error 4 in ld-2.17.so[7f05e0f29000+22000]
7[10434]: segfault at 8 ip 00007fad57f947e8 sp 00007ffe97b8afc0 error 4 in ld-2.17.so[7fad57f89000+22000]
7[11652]: segfault at 8 ip 00007f732250c7e8 sp 00007ffc53abd050 error 4 in ld-2.17.so[7f7322501000+22000]
1[12945]: segfault at 8 ip 00007f8aa96f57e8 sp 00007ffd18726110 error 4 in ld-2.17.so[7f8aa96ea000+22000]
9[13459]: segfault at 8 ip 00007f0a736f97e8 sp 00007ffcc0b68770 error 4 in ld-2.17.so[7f0a736ee000+22000]
LustreError: 13680:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000404:0x3c8b:0x0] mdc close failed: rc = -13
LustreError: 13680:0:(file.c:248:ll_close_inode_openhandle()) Skipped 3 previous similar messages
19[19186]: segfault at 4017b0 ip 00000000004017b0 sp 00007fff6f5118d0 error 14
18[20488]: segfault at 8 ip 00007fe5c70027e8 sp 00007ffcee213090 error 4 in ld-2.17.so[7fe5c6ff7000+22000]
12[21324]: segfault at 8 ip 00007f3b5c5a37e8 sp 00007ffee49f71b0 error 4 in ld-2.17.so[7f3b5c598000+22000]
0[21998]: segfault at 8 ip 00007f61f00777e8 sp 00007ffe895f7350 error 4 in ld-2.17.so[7f61f006c000+22000]
5[27525]: segfault at 0 ip (null) sp 00007ffd5a7be078 error 14 in 5[400000+6000]
7[816]: segfault at 8 ip 00007f3033c6d7e8 sp 00007ffe142ce210 error 4 in ld-2.17.so[7f3033c62000+22000]
0[9849]: segfault at 8 ip 00007f8f76e957e8 sp 00007ffd4f912dc0 error 4 in ld-2.17.so[7f8f76e8a000+22000]
Lustre: 15332:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x474f:0x0] with magic=0xbd60bd0
Lustre: 15332:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 21 previous similar messages
16[19349]: segfault at 8 ip 00007f9b259517e8 sp 00007ffffd5e6f80 error 4 in ld-2.17.so[7f9b25946000+22000]
13[31229]: segfault at 8 ip 00007fbde0c557e8 sp 00007ffcad986c90 error 4 in ld-2.17.so[7fbde0c4a000+22000]
18[6834]: segfault at 0 ip 0000000000401898 sp 00007fff774e4fa8 error 6 in 18[400000+6000]
ptlrpc_watchdog_fire: 2 callbacks suppressed
Lustre: mdt00_004: service thread pid 13028 was inactive for 40.131 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 9644, comm: mdt00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 11851, comm: mdt00_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 4 previous similar messages
Pid: 13028, comm: mdt00_004 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_link+0x8fb/0xff0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0003-osc-ffff8802ed4c37e8: disconnect after 21s idle
LustreError: 9314:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802d2764b40/0xd06913fc90292376 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x4062:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd06913fc9029233e expref: 1313 pid: 16297 timeout: 1808 lvb_type: 0
Lustre: mdt00_004: service thread pid 13028 completed after 100.309s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 11851 completed after 100.296s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802ed4c37e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802ed4c37e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802ed4c37e8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_001: service thread pid 9644 completed after 100.313s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 8644:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000405:0x4062:0x0] error: rc = -5
LustreError: 8644:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 38 previous similar messages
LustreError: 8357:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0000-mdc-ffff8802ed4c37e8: [0x200000401:0x1:0x0] lock enqueue fails: rc = -108
LustreError: 8357:0:(mdc_request.c:1470:mdc_read_page()) Skipped 18 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8802ed4c37e8: Connection restored to 192.168.123.47@tcp (at 0@lo)
14[26923]: segfault at 8 ip 00007f34a36be7e8 sp 00007fffd8870de0 error 4 in ld-2.17.so[7f34a36b3000+22000]
Lustre: 15830:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 588, rollback = 7
Lustre: 15830:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 579 previous similar messages
Lustre: 15830:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15830:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 15830:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15830:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 15830:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/588/0, punch: 0/0/0, quota 1/3/0
Lustre: 15830:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 15830:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15830:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 579 previous similar messages
Lustre: 15830:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15830:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 579 previous similar messages
19[13293]: segfault at 8 ip 00007fe9af9267e8 sp 00007ffed6402060 error 4 in ld-2.17.so[7fe9af91b000+22000]
16[14997]: segfault at 8 ip 00007f820dbff7e8 sp 00007fff62500360 error 4 in ld-2.17.so[7f820dbf4000+22000]
LustreError: 22404:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ed4c37e8: inode [0x200000406:0xf5e:0x0] mdc close failed: rc = -13
LustreError: 22404:0:(file.c:248:ll_close_inode_openhandle()) Skipped 13 previous similar messages
12[23151]: segfault at 8 ip 00007fd6853d17e8 sp 00007ffe44448640 error 4 in ld-2.17.so[7fd6853c6000+22000]
4[16647]: segfault at 8 ip 00007f1c5cb417e8 sp 00007ffe6477d230 error 4 in ld-2.17.so[7f1c5cb36000+22000]
10[16965]: segfault at 8 ip 00007f3e5a90d7e8 sp 00007ffd065e7a80 error 4 in ld-2.17.so[7f3e5a902000+22000]
3[24452]: segfault at 0 ip 0000000000403e5f sp 00007ffd572f2c50 error 6 in 3[400000+6000]
13[31291]: segfault at 8 ip 00007f7792f047e8 sp 00007ffc77937ef0 error 4 in ld-2.17.so[7f7792ef9000+22000]
18[32388]: segfault at 8 ip 00007f7eec2397e8 sp 00007ffd99809ed0 error 4 in ld-2.17.so[7f7eec22e000+22000]
0[32405]: segfault at 0 ip 0000000000403e5f sp 00007ffee30996b0 error 6 in 0[400000+6000]
8[2002]: segfault at 0 ip (null) sp 00007ffd3612f468 error 14 in 8[400000+6000]
8[2170]: segfault at 8 ip 00007fa633e1f7e8 sp 00007ffdbfbc2480 error 4 in ld-2.17.so[7fa633e14000+22000]
16[12463]: segfault at 8 ip 00007fa8b42a27e8 sp 00007fff627654a0 error 4 in ld-2.17.so[7fa8b4297000+22000]
Lustre: 31285:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x7546:0x0] with magic=0xbd60bd0
Lustre: 31285:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 29 previous similar messages
8[28849]: segfault at 8 ip 00007f0bae2c47e8 sp 00007ffe52262c50 error 4 in ld-2.17.so[7f0bae2b9000+22000]
5[30286]: segfault at 8 ip 00007f0e321957e8 sp 00007ffef8fa5290 error 4 in ld-2.17.so[7f0e3218a000+22000]
17[32635]: segfault at 8 ip 00007fea0e2817e8 sp 00007ffc8376fab0 error 4 in ld-2.17.so[7fea0e276000+22000]
18[5098]: segfault at 8 ip 00007f93e65a97e8 sp 00007ffca0b1eb20 error 4 in ld-2.17.so[7f93e659e000+22000]
19[14197]: segfault at 8 ip 00007fbd4b9247e8 sp 00007fff85c3abe0 error 4 in ld-2.17.so[7fbd4b919000+22000]
13[15315]: segfault at 0 ip 0000000000403e5f sp 00007ffdb5620260 error 6 in 13[400000+6000]
5[16088]: segfault at 8 ip 00007f3a57b0e7e8 sp 00007ffd56b1ca50 error 4 in ld-2.17.so[7f3a57b03000+22000]
16[16676]: segfault at 8 ip 00007f24137327e8 sp 00007ffd3e6b16d0 error 4 in ld-2.17.so[7f2413727000+22000]
13[29451]: segfault at 8 ip 00007f72c5efe7e8 sp 00007ffddc8f6bb0 error 4 in ld-2.17.so[7f72c5ef3000+22000]
18[2554]: segfault at 8 ip 00007fe24cd307e8 sp 00007ffcbee86630 error 4 in ld-2.17.so[7fe24cd25000+22000]
19[14605]: segfault at 8 ip 00007f99d472e7e8 sp 00007ffeba939e60 error 4 in ld-2.17.so[7f99d4723000+22000]
6[29622]: segfault at 8 ip 00007f4afd8377e8 sp 00007ffdfb168080 error 4 in ld-2.17.so[7f4afd82c000+22000]
LustreError: 4049:0:(statahead.c:2457:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
13[3982]: segfault at 0 ip 0000000000403e5f sp 00007ffcdc1f8360 error 6 in 13[400000+6000]
10[6986]: segfault at 406000 ip 0000000000406000 sp 00007ffff0f37ec8 error 14 in 10[606000+1000]
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 16270:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 689 previous similar messages
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16270:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 689 previous similar messages
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 16270:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 689 previous similar messages
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 16270:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 689 previous similar messages
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16270:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 689 previous similar messages
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16270:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 689 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2a1592067 PUD 2bb7c2067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata
CPU: 2 PID: 7567 Comm: ll_sa_7555 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802d4e69280 ti: ffff8802c6748000 task.ti: ffff8802c6748000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802c674bd78 EFLAGS: 00010296
RAX: ffff880081d7e990 RBX: 0000000000000008 RCX: 0000000100260016
RDX: 0000000000000026 RSI: ffff880081d7ecc0 RDI: 0000000000000008
RBP: ffff8802c674bd88 R08: ffff88009e1f48e8 R09: 0000000000000000
R10: ffff88009e1f6a08 R11: ffff88009e1f48e8 R12: 0000000000000000
R13: ffff88009a92dc40 R14: ffff880081d7ecc0 R15: ffff88009e1f48e8
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002acb2c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15b5791>] ll_statahead_thread+0xdd1/0x2020 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa15b49c0>] ? ll_statahead_handle.constprop.30+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 12321:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d2387840 x1837034453014656/t4294969148(0) o101->63098b92-42e1-4fb3-91f4-4408e3e09d3b@0@lo:455/0 lens 384/840 e 0 to 0 dl 1751932655 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: mdt_io00_003: service thread pid 23031 was inactive for 40.084 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16667, comm: mdt00_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1854/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 1 previous similar message
Pid: 23031, comm: mdt_io00_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0x57/0xf0 [mdt]
[<0>] mdt_reint_rename+0xde7/0x3ae0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_012: service thread pid 23866 was inactive for 40.138 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: mdt00_002: service thread pid 12322 was inactive for 40.137 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_006: service thread pid 22214 was inactive for 40.122 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Pid: 23866, comm: mdt00_012 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xc5a/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 11895:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802d3281e40/0xd0ff2d47ef8f1c69 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xcb:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd0ff2d47ef8f1c4d expref: 87 pid: 23842 timeout: 207 lvb_type: 0
LustreError: 18271:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a3c46fc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d2b170c0/0xd0ff2d47ef8f1ef4 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 13 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd0ff2d47ef8f1ee6 expref: 19 pid: 18271 timeout: 0 lvb_type: 0
Lustre: mdt_io00_003: service thread pid 23031 completed after 100.515s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 18271 completed after 99.271s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 12321 completed after 99.281s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_008: service thread pid 22478 completed after 99.260s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 12322 completed after 99.294s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 23831 completed after 99.283s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_012: service thread pid 23866 completed after 99.298s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 23842 completed after 99.265s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 16667 completed after 100.457s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 22214 completed after 99.284s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 27960:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 27552:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 28494:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000402:0xa8:0x0] mdc close failed: rc = -108
LustreError: 27388:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000401:0xcb:0x0] error -108.
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection restored to 192.168.123.57@tcp (at 0@lo)
Lustre: 28692:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x18e:0x0] with magic=0xbd60bd0
LustreError: 18271:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x1d1:0x0] ACL: rc = -2
LustreError: 9619:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000403:0x1ca:0x0] mdc close failed: rc = -13
LustreError: 9619:0:(file.c:248:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 14479:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000403:0x343:0x0] mdc close failed: rc = -13
10[24697]: segfault at 8 ip 00007f52b145e7e8 sp 00007fffd9712ac0 error 4 in ld-2.17.so[7f52b1453000+22000]
9[28407]: segfault at 8 ip 00007f7940fc97e8 sp 00007ffde1150230 error 4 in ld-2.17.so[7f7940fbe000+22000]
LustreError: 32304:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000402:0x66a:0x0] mdc close failed: rc = -13
LustreError: 22214:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x6ab:0x0] ACL: rc = -2
LustreError: 23872:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x6a8:0x0] ACL: rc = -2
0[7589]: segfault at 0 ip (null) sp 00007ffc6c0ce268 error 14 in 0[400000+6000]
LustreError: 22041:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802da8fb7e8: inode [0x200000402:0x97e:0x0] mdc close failed: rc = -13
Lustre: 16667:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xc90:0x0] with magic=0xbd60bd0
Lustre: 16667:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 26063:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802da8fb7e8: inode [0x200000402:0xed7:0x0] mdc close failed: rc = -13
15[32086]: segfault at 8 ip 00007f1a932db7e8 sp 00007ffeff1fb0f0 error 4 in ld-2.17.so[7f1a932d0000+22000]
14[10258]: segfault at 4017b0 ip 00000000004017b0 sp 00007fff397bd010 error 14
LustreError: 10258:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000402:0x1103:0x0] mdc close failed: rc = -13
Lustre: 28708:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1065:0x0] with magic=0xbd60bd0
Lustre: 28708:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 5034:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802da8fb7e8: inode [0x200000403:0x13dd:0x0] mdc close failed: rc = -13
LustreError: 5034:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 23842:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x14ff:0x0] with magic=0xbd60bd0
Lustre: 23842:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
12[25236]: segfault at 8 ip 00007f79993697e8 sp 00007fffef8a1830 error 4 in ld-2.17.so[7f799935e000+22000]
19[31338]: segfault at 8 ip 00007f9005f237e8 sp 00007ffce19c8ce0 error 4 in ld-2.17.so[7f9005f18000+22000]
LustreError: 18743:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802da8fb7e8: inode [0x200000403:0x19a8:0x0] mdc close failed: rc = -13
19[27279]: segfault at 0 ip 0000000000403e5f sp 00007fff3b8d47e0 error 6 in 19[400000+6000]
11[25128]: segfault at 0 ip (null) sp 00007fff375080a8 error 14 in 11[400000+6000]
Lustre: 12319:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2417:0x0] with magic=0xbd60bd0
Lustre: 12319:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 28708:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x25e9:0x0] with magic=0xbd60bd0
Lustre: 28708:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 10053:0:(statahead.c:2457:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: 27131:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2709:0x0] with magic=0xbd60bd0
Lustre: 27131:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
5[23960]: segfault at 8 ip 00007f1a10c697e8 sp 00007ffebb7b1510 error 4 in ld-2.17.so[7f1a10c5e000+22000]
6[712]: segfault at 8 ip 00007f858351e7e8 sp 00007ffc2ceaced0 error 4 in ld-2.17.so[7f8583513000+22000]
LustreError: 18271:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x29a6:0x0] ACL: rc = -2
LustreError: 23872:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x2a8d:0x0] ACL: rc = -2
ptlrpc_watchdog_fire: 6 callbacks suppressed
Lustre: mdt00_013: service thread pid 23872 was inactive for 40.002 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 5 previous similar messages
Lustre: mdt00_009: service thread pid 23831 was inactive for 40.000 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 23831, comm: mdt00_009 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 23842, comm: mdt00_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 28708, comm: mdt00_019 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1739/0x2d20 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_005: service thread pid 22127 was inactive for 40.096 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 11895:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802c6f77840/0xd0ff2d47efe96a36 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x2cab:0x0].0x0 bits 0x1b/0x0 rrc: 11 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd0ff2d47efe96a1a expref: 1102 pid: 23872 timeout: 818 lvb_type: 0
Lustre: mdt00_013: service thread pid 23872 completed after 100.185s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 28669 completed after 99.935s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_019: service thread pid 28708 completed after 99.932s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 23842 completed after 100.003s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802da8fb7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8802da8fb7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802da8fb7e8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_009: service thread pid 23831 completed after 100.197s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 23002:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802da8fb7e8: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
LustreError: 23002:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: mdt00_002: service thread pid 12322 completed after 99.869s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 21329:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 21329:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 10 previous similar messages
Lustre: mdt00_012: service thread pid 23866 completed after 99.858s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_002: service thread pid 12354 completed after 99.558s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_020: service thread pid 27131 completed after 99.049s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 22127 completed after 99.061s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_008: service thread pid 22478 completed after 99.064s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8802da8fb7e8: Connection restored to 192.168.123.57@tcp (at 0@lo)
Lustre: 22478:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x9d:0x0] with magic=0xbd60bd0
Lustre: 22478:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 3 previous similar messages
15[12639]: segfault at 8 ip 00007f99d24d47e8 sp 00007ffdad6f3540 error 4 in ld-2.17.so[7f99d24c9000+22000]
7[18034]: segfault at 0 ip 0000000000403e5f sp 00007ffeb38b4d70 error 6 in 7[400000+6000]
LustreError: 28708:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x4fa:0x0] ACL: rc = -2
16[1130]: segfault at 8 ip 00007f2858ff87e8 sp 00007ffe41974990 error 4 in ld-2.17.so[7f2858fed000+22000]
Lustre: 22465:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x315b:0x0] with magic=0xbd60bd0
Lustre: 22465:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
3[2557]: segfault at 8 ip 00007fd32f8b47e8 sp 00007ffd8ff02240 error 4 in ld-2.17.so[7fd32f8a9000+22000]
LustreError: 11895:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802b31ae580/0xd0ff2d47effccba0 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x3481:0x0].0x0 bits 0x13/0x0 rrc: 17 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd0ff2d47effccb92 expref: 1007 pid: 23866 timeout: 1041 lvb_type: 0
LustreError: 28706:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802c3e88958 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b77243c0/0xd0ff2d47effcf0f3 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x3481:0x0].0x0 bits 0x1b/0x0 rrc: 12 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd0ff2d47effcf0bb expref: 192 pid: 28706 timeout: 0 lvb_type: 0
LustreError: 11888:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1751933583 with bad export cookie 15059805465935290478
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 28706:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 22694:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x3481:0x0] error: rc = -5
LustreError: 22694:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 9 previous similar messages
LustreError: 22278:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: Skipped 3 previous similar messages
LustreError: 22337:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000403:0x3481:0x0] error -108.
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection restored to 192.168.123.57@tcp (at 0@lo)
Lustre: lustre-OST0000-osc-ffff8802db315d28: disconnect after 24s idle
LustreError: 11895:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88009dce7840/0xd0ff2d47effd45e8 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xd0ff2d47effd457f expref: 47 pid: 12335 timeout: 1143 lvb_type: 0
LustreError: 27131:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802b60ba548 ns: mdt-lustre-MDT0000_UUID lock: ffff8802b6258b80/0xd0ff2d47effd51b8 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 20 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xd0ff2d47effd51aa expref: 17 pid: 27131 timeout: 0 lvb_type: 0
LustreError: 27131:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802db315d28: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
LustreError: 24785:0:(file.c:6198:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 24785:0:(file.c:6198:ll_inode_revalidate_fini()) Skipped 41 previous similar messages
LustreError: 25492:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802db315d28: inode [0x200000404:0x876:0x0] mdc close failed: rc = -108
LustreError: 25492:0:(file.c:248:ll_close_inode_openhandle()) Skipped 18 previous similar messages
LustreError: 25492:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8802db315d28: namespace resource [0x200000401:0x1:0x0].0x0 (ffff8800a14c5440) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff8802db315d28: Connection restored to 192.168.123.57@tcp (at 0@lo)
10[10993]: segfault at 8 ip 00007f78fee337e8 sp 00007ffd92925d70 error 4 in ld-2.17.so[7f78fee28000+22000]
Lustre: 12322:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0xefa:0x0] with magic=0xbd60bd0
Lustre: 12322:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 5600:0:(statahead.c:2409:start_statahead_thread()) lustre: invalid pattern 0X0.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 31e784067 PUD 31b2b4067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 5 PID: 13445 Comm: ll_sa_13196 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802ef523760 ti: ffff8802cd8dc000 task.ti: ffff8802cd8dc000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802cd8dfd78 EFLAGS: 00010296
RAX: ffff88029612e990 RBX: 0000000000000008 RCX: 0000000100260022
RDX: 0000000000000026 RSI: ffff88029612ecc0 RDI: 0000000000000008
RBP: ffff8802cd8dfd88 R08: ffff880325051dd8 R09: 0000000000000000
R10: ffff880325054c38 R11: ffff880325051dd8 R12: 0000000000000000
R13: ffff8802ef525c40 R14: ffff88029612ecc0 R15: ffff880325051dd8
FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002cf4cc000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1786791>] ll_statahead_thread+0xdd1/0x2020 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa17859c0>] ? ll_statahead_handle.constprop.30+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

LustreError: 30137:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e8254a88: inode [0x200000401:0x11:0x0] mdc close failed: rc = -13
Lustre: 29743:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x2e:0x0] with magic=0xbd60bd0
12[376]: segfault at 8 ip 00007f4ea0cff7e8 sp 00007ffc259a4c70 error 4 in ld-2.17.so[7f4ea0cf4000+22000]
LustreError: 2022:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e8254a88: inode [0x200000401:0x142:0x0] mdc close failed: rc = -13
Lustre: 1007:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x1df:0x0] with magic=0xbd60bd0
Lustre: 1007:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 27849:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 27849:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27849:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27849:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 27849:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27849:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
9[4294]: segfault at 8 ip 00007f64a3b827e8 sp 00007ffeed9f1e50 error 4 in ld-2.17.so[7f64a3b77000+22000]
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 27849:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 27849:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27849:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27849:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 27849:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27849:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27849:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 3 previous similar messages
19[6889]: segfault at 8 ip 00007fae0e2cd7e8 sp 00007ffced8d4370 error 4 in ld-2.17.so[7fae0e2c2000+22000]
Lustre: 29946:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802efe63740 x1836925584035456/t4294978939(0) o101->65758477-e7bf-4e52-8671-5c8a9053fefc@0@lo:77/0 lens 376/816 e 0 to 0 dl 1751828842 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 27849:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 27849:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27849:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27849:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/78/0
Lustre: 27849:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27849:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27849:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27849:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 27848:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 27848:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 27848:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27848:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 27848:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27848:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 27848:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 27848:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 27848:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27848:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 27848:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27848:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 9088:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e8254a88: inode [0x200000401:0x3a9:0x0] mdc close failed: rc = -13
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 19 previous similar messages
LustreError: 12334:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e8254a88: inode [0x200000401:0x4cb:0x0] mdc close failed: rc = -13
5[12386]: segfault at 8 ip 00007fec166967e8 sp 00007ffd43408b60 error 4 in ld-2.17.so[7fec1668b000+22000]
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 30044:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 30044:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 30044:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 30044:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 30044:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 30044:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1268:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x551:0x0] with magic=0xbd60bd0
Lustre: 1268:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 27577:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x72f:0x0] with magic=0xbd60bd0
Lustre: 27577:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 761:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 761:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 761:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 761:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 761:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 761:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 761:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 761:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 761:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 761:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 761:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 761:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 39 previous similar messages
12[24591]: segfault at 8 ip 00007fa112d927e8 sp 00007ffe74a0f570 error 4 in ld-2.17.so[7fa112d87000+22000]
7[24931]: segfault at 8 ip 00007f50fc9947e8 sp 00007ffeafe25af0 error 4 in ld-2.17.so[7f50fc989000+22000]
1[24992]: segfault at 8 ip 00007f2b081b37e8 sp 00007fff70ed5b90 error 4 in ld-2.17.so[7f2b081a8000+22000]
8[25129]: segfault at 0 ip (null) sp 00007ffc7f0ae6a8 error 14 in 8[400000+6000]
1[26889]: segfault at 8 ip 00007fc2b0c137e8 sp 00007ffecb67e600 error 4 in ld-2.17.so[7fc2b0c08000+22000]
LustreError: 26889:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800ad552548: inode [0x200000402:0x98a:0x0] mdc close failed: rc = -13
LustreError: 26889:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
9[27082]: segfault at 8 ip 00007f30f01067e8 sp 00007ffe8fb20110 error 4 in ld-2.17.so[7f30f00fb000+22000]
Lustre: 1268:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xb7b:0x0] with magic=0xbd60bd0
Lustre: 1268:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
3[1873]: segfault at 8 ip 00007f7c9d4557e8 sp 00007fffb2243150 error 4 in ld-2.17.so[7f7c9d44a000+22000]
5[3091]: segfault at 8 ip 00007fec052e37e8 sp 00007ffc9bd37a30 error 4 in ld-2.17.so[7fec052d8000+22000]
12[5505]: segfault at 8 ip 00007f4b0b1ad7e8 sp 00007ffc4f4d6050 error 4 in ld-2.17.so[7f4b0b1a2000+22000]
Lustre: 28372:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xe61:0x0] with magic=0xbd60bd0
Lustre: 28372:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 29811:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 29811:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 83 previous similar messages
Lustre: 29811:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 29811:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 83 previous similar messages
Lustre: 29811:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 29811:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 83 previous similar messages
Lustre: 29811:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 29811:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 83 previous similar messages
Lustre: 29811:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 29811:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 83 previous similar messages
Lustre: 29811:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 29811:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 83 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata
CPU: 15 PID: 28710 Comm: ll_sa_28686 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802d480b760 ti: ffff88029aabc000 task.ti: ffff88029aabc000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88029aabfd78 EFLAGS: 00010296
RAX: ffff88027b4be990 RBX: 0000000000000008 RCX: 0000000100260013
RDX: 0000000000000026 RSI: ffff88027b4becc0 RDI: 0000000000000008
RBP: ffff88029aabfd88 R08: ffff8802d1876018 R09: 0000000000000000
R10: ffff8802d1873858 R11: ffff8802d1876018 R12: 0000000000000000
R13: ffff8802d8f449d0 R14: ffff88027b4becc0 R15: ffff8802d1876018
FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15da831>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa15d9a80>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 22726:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x4c:0x0] with magic=0xbd60bd0
Lustre: 22726:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d7952d40 x1835661204151552/t4294971925(0) o101->686b1cd5-4361-4331-90b1-70102095b23e@0@lo:21/0 lens 376/864 e 0 to 0 dl 1750623051 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
8[3373]: segfault at 8 ip 00007fae79d057e8 sp 00007ffff33399a0 error 4 in ld-2.17.so[7fae79cfa000+22000]
9[5285]: segfault at 8 ip 00007fe4749287e8 sp 00007fff3ed68ba0 error 4 in ld-2.17.so[7fe47491d000+22000]
19[9727]: segfault at 8 ip 00007fcbca9827e8 sp 00007ffdd20ee380 error 4 in ld-2.17.so[7fcbca977000+22000]
LustreError: 26437:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x44b:0x0] ACL: rc = -2
LustreError: 11576:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x59f:0x0] ACL: rc = -2
13[22412]: segfault at 8 ip 00007f5b699467e8 sp 00007ffeb42e5d90 error 4 in ld-2.17.so[7f5b6993b000+22000]
19[26364]: segfault at 8 ip 00007feb564ca7e8 sp 00007ffd64439290 error 4 in ld-2.17.so[7feb564bf000+22000]
LustreError: 30401:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a28712a8: inode [0x200000402:0x8cf:0x0] mdc close failed: rc = -13
LustreError: 22693:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0xb63:0x0] ACL: rc = -2
LustreError: 25905:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a28712a8: inode [0x200000402:0xcf5:0x0] mdc close failed: rc = -13
1[7468]: segfault at 8 ip 00007f33805e27e8 sp 00007ffdb6bdbab0 error 4 in ld-2.17.so[7f33805d7000+22000]
6[29367]: segfault at 8 ip 00007f352582d7e8 sp 00007ffe75d73df0 error 4 in ld-2.17.so[7f3525822000+22000]
LustreError: 30916:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a28712a8: inode [0x200000402:0x12d2:0x0] mdc close failed: rc = -13
1[6116]: segfault at 8 ip 00007f9199c837e8 sp 00007ffcffb37500 error 4 in ld-2.17.so[7f9199c78000+22000]
LustreError: 26477:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x1630:0x0] ACL: rc = -2

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD a3275067 PUD a0503067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks libata virtio_blk serio_raw i2c_core floppy
CPU: 11 PID: 20234 Comm: ll_sa_20144 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88029ed9c9d0 ti: ffff88029dc08000 task.ti: ffff88029dc08000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88029dc0bd78 EFLAGS: 00010296
RAX: ffff88008d6af310 RBX: 0000000000000008 RCX: 0000000100260018
RDX: 0000000000000026 RSI: ffff88008d6af640 RDI: 0000000000000008
RBP: ffff88029dc0bd88 R08: ffff8802de5431b8 R09: 0000000000000000
R10: ffff8802de546368 R11: ffff8802de5431b8 R12: 0000000000000000
R13: ffff8802b27c8010 R14: ffff88008d6af640 R15: ffff8802de5431b8
FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002d36a2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1127831>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa1126a80>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 10011:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 510 < left 922, rollback = 2
Lustre: 10011:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/3, destroy: 1/4/0
Lustre: 10011:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 13/922/0
Lustre: 10011:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 10011:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 15/263/2, delete: 3/6/0
Lustre: 10011:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 3/3/0
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 507 < left 610, rollback = 2
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 8/610/0
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 12/275/4, delete: 0/0/0
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 1100, rollback = 2
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/1, destroy: 1/4/0
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 15/1100/0
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/129/0
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 15/263/2, delete: 3/6/0
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 3/3/0
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 11 previous similar messages
1[20543]: segfault at 8 ip 00007fb688e057e8 sp 00007ffde6128690 error 4 in ld-2.17.so[7fb688dfa000+22000]
LustreError: 10009:0:(mdd_dir.c:4747:mdd_migrate_cmd_check()) lustre-MDD0001: '19' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 19' to finish migration: rc = -1
LustreError: 10009:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x2:0x0]/19 failed: rc = -1
Lustre: 18724:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/17 is open, migrate only dentry
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 788, rollback = 2
Lustre: 10009:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 23 previous similar messages
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 10009:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 10009:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 10009:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 11/239/3, delete: 0/0/0
Lustre: 10009:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 10009:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 11846:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
LustreError: 9992:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0xc6:0x0]: rc = -2
LustreError: 21300:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e77b92a8: inode [0x200000404:0xc6:0x0] mdc close failed: rc = -2
Lustre: 17922:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c9dea340 x1835060326492800/t4294968612(0) o101->c2b9d73d-6a44-4224-b35a-6b1a10263169@0@lo:31/0 lens 376/840 e 0 to 0 dl 1750050016 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 10009:0:(mdd_dir.c:4747:mdd_migrate_cmd_check()) lustre-MDD0002: '12' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 12' to finish migration: rc = -1
LustreError: 10009:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000403:0x1:0x0]/12 failed: rc = -1
LustreError: 9992:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0xc6:0x0]: rc = -2
LustreError: 17085:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x200000404:0xc6:0x0] mdc close failed: rc = -2
Lustre: 22591:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/1 is open, migrate only dentry
Lustre: 11846:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 11846:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11846:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11846:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 41 previous similar messages
Lustre: 11846:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11846:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 41 previous similar messages
Lustre: 11846:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 11846:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 41 previous similar messages
Lustre: 11846:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11846:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 41 previous similar messages
Lustre: 11846:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11846:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 41 previous similar messages
Lustre: 22130:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 512 < left 652, rollback = 2
Lustre: 22130:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 10011:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 10011:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 10011:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 10011:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 10011:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 10011:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 10011:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 10011:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 10011:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 10011:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: dir [0x240000404:0x66:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 509 < left 936, rollback = 2
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 19 previous similar messages
LustreError: 27547:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0xbc:0x0]: rc = -5
LustreError: 27547:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 22136:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802efb56940 x1835060331133568/t4294971905(0) o101->2c2a4d20-f6e5-4652-9661-2dfed01992b8@0@lo:75/0 lens 376/864 e 0 to 0 dl 1750050060 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 19060:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 19060:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 19060:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 19060:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 19060:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 19060:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 19060:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 19060:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 19060:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 19060:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 19060:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 19060:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 13347:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802cd31e440 x1835060331642752/t4294970525(0) o101->c2b9d73d-6a44-4224-b35a-6b1a10263169@0@lo:78/0 lens 376/816 e 0 to 0 dl 1750050063 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 509 < left 922, rollback = 2
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 28 previous similar messages
LustreError: 31792:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x141:0x0]: rc = -5
LustreError: 31792:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 32388:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x141:0x0]: rc = -5
LustreError: 32388:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 26521:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x200000403:0x163:0x0] mdc close failed: rc = -13
18[1954]: segfault at 8 ip 00007f46eb2d37e8 sp 00007ffd17c963a0 error 4 in ld-2.17.so[7f46eb2c8000+22000]
Lustre: 18724:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/3 is open, migrate only dentry
Lustre: 18724:0:(mdd_dir.c:4826:mdd_migrate_object()) Skipped 1 previous similar message
Lustre: 22428:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 510 < left 699, rollback = 2
Lustre: 22428:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 14 previous similar messages
Lustre: 22428:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 22428:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 22428:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 22428:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 22428:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 22428:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 22428:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0
Lustre: 22428:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 22428:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 22428:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 11846:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 11846:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
0[17185]: segfault at 8 ip 00007fe71f09e7e8 sp 00007fffaa0ad730 error 4 in ld-2.17.so[7fe71f093000+22000]
16[17317]: segfault at 8 ip 00007fa1fc4467e8 sp 00007fffcbdac170 error 4 in ld-2.17.so[7fa1fc43b000+22000]
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 618, rollback = 7
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
1[21011]: segfault at 8 ip 00007fa56cb1f7e8 sp 00007ffdf84ba450 error 4 in ld-2.17.so[7fa56cb14000+22000]
Lustre: 10011:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/3 is open, migrate only dentry
Lustre: 10011:0:(mdd_dir.c:4826:mdd_migrate_object()) Skipped 1 previous similar message
11[24014]: segfault at 8 ip 00007f550442c7e8 sp 00007ffc2876aca0 error 4 in ld-2.17.so[7f5504421000+22000]
6[25703]: segfault at 1c ip 00000000004017b0 sp 00007fffc8c0be00 error 6 in 6[400000+6000]
4[25898]: segfault at 8 ip 00007f9f31d6e7e8 sp 00007fff07659a70 error 4 in ld-2.17.so[7f9f31d63000+22000]
8[25868]: segfault at 8 ip 00007f95638be7e8 sp 00007ffdee2313f0 error 4 in ld-2.17.so[7f95638b3000+22000]
Lustre: 11845:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 11845:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 10011:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 17176:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x73d:0x0] with magic=0xbd60bd0
1[30863]: segfault at 0 ip 0000000000403f00 sp 00007ffd0a601978 error 6 in 1[400000+6000]
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 618, rollback = 7
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 15904:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15904:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 15904:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 15904:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 15904:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15904:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 15904:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15904:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 15904:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15904:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 10011:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/15 is open, migrate only dentry
INFO: task mkdir:30510 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8800a4390880 11616 30510 16669 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:31372 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8800a6db8880 13056 31372 16737 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:301 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802e1534240 13056 301 16749 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802db431878 13056 355 16528 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
Lustre: 10011:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 509 < left 610, rollback = 2
Lustre: 10011:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15904:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
6[5253]: segfault at 0 ip 00000000004043f0 sp 00007fffee734bb8 error 6 in 6[400000+6000]
LustreError: 22428:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/12 failed: rc = -2
LustreError: 22428:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 1 previous similar message
8[8716]: segfault at 8 ip 00007f8c472217e8 sp 00007ffdd8ba2f00 error 4 in ld-2.17.so[7f8c47216000+22000]
12[11820]: segfault at 8 ip 00007f37b46dd7e8 sp 00007fffa318b6f0 error 4 in ld-2.17.so[7f37b46d2000+22000]
10[18722]: segfault at 8 ip 00007f4266d887e8 sp 00007ffe0e252d00 error 4 in ld-2.17.so[7f4266d7d000+22000]
9[21960]: segfault at 8 ip 00007f49b75077e8 sp 00007fff3dfd5990 error 4 in ld-2.17.so[7f49b74fc000+22000]
9[22383]: segfault at 8 ip 00007fc6588567e8 sp 00007ffdee09bc90 error 4 in ld-2.17.so[7fc65884b000+22000]
5[22804]: segfault at 406000 ip 0000000000406000 sp 00007fff56a6f538 error 14 in 5[606000+1000]
INFO: task mkdir:30117 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8800a339aae8 13056 30117 16564 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:30128 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff880096834fc8 13056 30128 16716 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:30197 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802decc2980 13056 30197 16575 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:30242 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802dfd32140 13024 30242 16568 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:32465 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802d4ebeb80 13056 32465 16917 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mkdir:32667 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff8802e1530880 13056 32667 16655 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81256f65>] filename_create+0x85/0x180
[<ffffffff81257c94>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257f41>] user_path_create+0x41/0x60
[<ffffffff81259286>] SyS_mkdirat+0x46/0xe0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff81259339>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
Lustre: 21519:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 21519:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 21 previous similar messages
Lustre: 21519:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 21519:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 21519:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 21519:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 21519:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 21519:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 21519:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 21519:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 21519:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21519:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 39 previous similar messages
4[1645]: segfault at 406000 ip 0000000000406000 sp 00007ffcbc5c7738 error 14 in 4 (deleted)[606000+1000]
9[3046]: segfault at 8 ip 00007fd7c0d477e8 sp 00007ffe775c0290 error 4 in ld-2.17.so[7fd7c0d3c000+22000]
10[3259]: segfault at 8 ip 00007ff9126c57e8 sp 00007ffcae1b33b0 error 4 in ld-2.17.so[7ff9126ba000+22000]
11[3830]: segfault at 0 ip (null) sp 00007fff3c1b1108 error 14 in 11[400000+6000]
2[3821]: segfault at 1c ip 00000000004017b0 sp 00007fff42ee72f0 error 6 in 2[400000+6000]
18[6437]: segfault at 8 ip 00007fbb886da7e8 sp 00007ffeb7f66b30 error 4 in ld-2.17.so[7fbb886cf000+22000]
Lustre: 10011:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 610, rollback = 2
Lustre: 10011:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 11 previous similar messages
LustreError: 8091:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x770:0x0]: rc = -5
LustreError: 8091:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
10[9679]: segfault at 404540 ip 0000000000404d7a sp 00007ffcbc2568b8 error 7 in 10[400000+6000]
9[12292]: segfault at 8 ip 00007f42895527e8 sp 00007ffc29dda610 error 4 in ld-2.17.so[7f4289547000+22000]
LustreError: 22130:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0xfdc:0x0]/5 failed: rc = -2
Lustre: 19060:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 19060:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 99 previous similar messages
LustreError: 25275:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x280000403:0x7c6:0x0] mdc close failed: rc = -13
LustreError: 25275:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
18[3532]: segfault at 8 ip 00007f77757fb7e8 sp 00007ffc219a44e0 error 4 in ld-2.17.so[7f77757f0000+22000]
17[3783]: segfault at 0 ip 0000000000403e5f sp 00007fff2127f700 error 6 in 17[400000+6000]
19[7439]: segfault at 8 ip 00007f883dd727e8 sp 00007ffcaddc5130 error 4 in ld-2.17.so[7f883dd67000+22000]
0[7429]: segfault at 8 ip 00007fac1ea737e8 sp 00007ffdf0dc44b0 error 4 in ld-2.17.so[7fac1ea68000+22000]
15[7635]: segfault at 8 ip 00007fa2789127e8 sp 00007ffc53ed6f60 error 4 in ld-2.17.so[7fa278907000+22000]
Lustre: 22428:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/0 is open, migrate only dentry
6[28054]: segfault at 0 ip (null) sp 00007ffcb3b25cc8 error 14 in 6[400000+6000]
5[30084]: segfault at 8 ip 00007f0d7502c7e8 sp 00007ffd0c502df0 error 4 in ld-2.17.so[7f0d75021000+22000]
11[32074]: segfault at c0cfcd ip 0000000000404768 sp 00007ffed9c70958 error 6
11[31987]: segfault at c0cfcd ip 0000000000404768 sp 00007ffe279dd6e8 error 6
in 11[400000+6000]
in 11[400000+6000]
14[32091]: segfault at 5468 ip 00000000004059a7 sp 00007fffdb45b7b8 error 6 in 14[400000+6000]
1[31854]: segfault at 0 ip 0000000000403e5f sp 00007ffce4fcacd0 error 6 in 1[400000+6000]
Lustre: 18724:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/17 is open, migrate only dentry
0[2516]: segfault at 8 ip 00007fc7b739a7e8 sp 00007ffe6f8649f0 error 4 in ld-2.17.so[7fc7b738f000+22000]
Lustre: 18283:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18283:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 183 previous similar messages
Lustre: 18283:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 18283:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 183 previous similar messages
Lustre: 18283:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 18283:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 183 previous similar messages
Lustre: 18283:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18283:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 183 previous similar messages
Lustre: 18283:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18283:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 183 previous similar messages
2[5743]: segfault at 8 ip 00007f48166b97e8 sp 00007ffc77f16650 error 4 in ld-2.17.so[7f48166ae000+22000]
0[10537]: segfault at 0 ip (null) sp 00007ffc914994d8 error 14 in 0[400000+6000]
16[24365]: segfault at 0 ip (null) sp 00007ffd2c2590f8 error 14 in 16[400000+6000]
10[25997]: segfault at 8 ip 00007f3a928817e8 sp 00007ffcba3c62c0 error 4 in ld-2.17.so[7f3a92876000+22000]
19[31176]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffc89ab0540 error 14
15[1985]: segfault at 0 ip (null) sp 00007ffe1fcb88b8 error 14 in 15[400000+6000]
17[7796]: segfault at 8 ip 00007fb08e68d7e8 sp 00007ffc0adecab0 error 4 in ld-2.17.so[7fb08e682000+22000]
Lustre: 20432:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x11b2:0x0] with magic=0xbd60bd0
Lustre: 20432:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 22428:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 22428:0:(mdd_dir.c:4826:mdd_migrate_object()) Skipped 1 previous similar message
9[12472]: segfault at 0 ip 0000000000403e5f sp 00007ffe22c9d380 error 6 in 7[400000+6000]
Lustre: 18283:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18283:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 165 previous similar messages
0[23995]: segfault at 8 ip 00007f67531127e8 sp 00007ffca079e470 error 4 in ld-2.17.so[7f6753107000+22000]
Lustre: 20432:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x133b:0x0] with magic=0xbd60bd0
Lustre: 20432:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 25331:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x280000404:0x1306:0x0] mdc close failed: rc = -13
Lustre: lustre-OST0001-osc-ffff8802e77b92a8: disconnect after 21s idle
Lustre: mdt00_019: service thread pid 21140 was inactive for 40.105 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 21140, comm: mdt00_019 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_reint_open+0x1274/0x2ed0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_intent_open+0x93/0x480 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 10011, comm: mdt_io00_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xabc/0xfd0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xb00 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x3c4/0x470 [mdt]
[<0>] mdt_rename_lock+0xd9/0x360 [mdt]
[<0>] mdt_reint_migrate+0x87e/0x23e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22389, comm: mdt_io00_007 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0x57/0xf0 [mdt]
[<0>] mdt_reint_migrate+0x1763/0x23e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd80 [ptlrpc]
Lustre: mdt00_007: service thread pid 17491 was inactive for 40.156 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io00_006: service thread pid 22374 was inactive for 40.047 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
Lustre: mdt00_005: service thread pid 13846 was inactive for 40.025 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: mdt00_010: service thread pid 18677 was inactive for 40.117 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 8 previous similar messages
Lustre: mdt_io00_003: service thread pid 18724 was inactive for 72.033 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
LustreError: 9825:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802bfe13880/0x6c82a01bb7e846e5 lrc: 3/0,0 mode: PW/PW res: [0x200000403:0x2304:0x0].0x0 bits 0x4/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x6c82a01bb7e846d0 expref: 1744 pid: 17610 timeout: 1001 lvb_type: 0
Lustre: mdt00_022: service thread pid 22069 completed after 96.469s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 22389:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 22389:0:(mdd_dir.c:4826:mdd_migrate_object()) Skipped 1 previous similar message
LustreError: 18677:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e6d892a8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802e2504000/0x6c82a01bb7e8ba5e lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1:0x0].0x0 bits 0x13/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x6c82a01bb7e8ba49 expref: 779 pid: 18677 timeout: 0 lvb_type: 0
Lustre: mdt00_013: service thread pid 19461 completed after 96.050s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_023: service thread pid 22136 completed after 95.801s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 17922 completed after 96.001s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 17176 completed after 97.127s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 20953 completed after 95.574s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_028: service thread pid 22682 completed after 97.803s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_018: service thread pid 21108 completed after 95.467s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802e77b92a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802e77b92a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_011: service thread pid 18687 completed after 96.002s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 9988 completed after 95.838s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_019: service thread pid 21140 completed after 100.206s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 17491 completed after 100.218s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_007: service thread pid 22389 completed after 100.215s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802e77b92a8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_003: service thread pid 13347 completed after 95.467s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_029: service thread pid 22692 completed after 98.684s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 18677 completed after 95.735s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 16211:0:(file.c:6195:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -5
Lustre: mdt_io00_002: service thread pid 10011 completed after 100.161s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 13846 completed after 97.711s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 13844 completed after 100.026s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 19245:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e77b92a8: inode [0x200000403:0x22f1:0x0] mdc close failed: rc = -108
Lustre: mdt_io00_011: service thread pid 22657 completed after 100.023s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 25485:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 25485:0:(llite_lib.c:2023:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 25444:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x22f1:0x0] error -108.
Lustre: mdt00_002: service thread pid 9990 completed after 95.548s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 15118:0:(mdc_request.c:1470:mdc_read_page()) lustre-MDT0000-mdc-ffff8802e77b92a8: [0x200000403:0x2:0x0] lock enqueue fails: rc = -108
Lustre: mdt_io00_008: service thread pid 22428 completed after 100.071s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 19184:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/11 failed: rc = -2
Lustre: mdt_io00_004: service thread pid 19184 completed after 99.870s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_005: service thread pid 22130 completed after 99.991s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8802e77b92a8: Connection restored to 192.168.123.62@tcp (at 0@lo)
Lustre: mdt_io00_006: service thread pid 22374 completed after 99.971s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_009: service thread pid 22503 completed after 99.849s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 504 < left 699, rollback = 2
Lustre: 18724:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 84 previous similar messages
Lustre: mdt_io00_003: service thread pid 18724 completed after 93.404s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_001: service thread pid 10010 completed after 93.080s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
hrtimer: interrupt took 13825468 ns
14[31022]: segfault at 8 ip 00007f43a7e877e8 sp 00007ffd6db12420 error 4 in ld-2.17.so[7f43a7e7c000+22000]
6[30890]: segfault at 1c ip 00000000004017b0 sp 00007ffc01d6f580 error 6 in 6[400000+6000]
4[32526]: segfault at 8 ip 00007f5334ff77e8 sp 00007ffdea96bff0 error 4 in ld-2.17.so[7f5334fec000+22000]
14[1426]: segfault at 8 ip 00007f65a43d67e8 sp 00007ffe96983970 error 4 in ld-2.17.so[7f65a43cb000+22000]
Lustre: 22657:0:(mdt_reint.c:2484:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x2:0x0]/13 is open, migrate only dentry
Lustre: 21157:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x15d3:0x0] with magic=0xbd60bd0
Lustre: 21157:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
0[18175]: segfault at 0 ip 0000000000403e5f sp 00007ffed781d580 error 6 in 0[400000+6000]
3[23345]: segfault at 8 ip 00007fcb7f3047e8 sp 00007ffd7e2708c0 error 4 in ld-2.17.so[7fcb7f2f9000+22000]
3[23482]: segfault at 8 ip 00007fe6dc50c7e8 sp 00007ffddb963060 error 4 in ld-2.17.so[7fe6dc501000+22000]
16[28379]: segfault at 8 ip 00007fd8484bf7e8 sp 00007ffd336430a0 error 4 in ld-2.17.so[7fd8484b4000+22000]
18[29090]: segfault at 8 ip 00007f6b2baef7e8 sp 00007fff57867800 error 4 in ld-2.17.so[7f6b2bae4000+22000]
7[30274]: segfault at 8 ip 00007f46d48277e8 sp 00007ffd0cb55100 error 4 in ld-2.17.so[7f46d481c000+22000]
18[31121]: segfault at 8 ip 00007fc48459b7e8 sp 00007ffd8d4184a0 error 4 in ld-2.17.so[7fc484590000+22000]
11[12697]: segfault at 8 ip 00007ff969c327e8 sp 00007ffe62a06f70 error 4 in ld-2.17.so[7ff969c27000+22000]
16[19540]: segfault at 8 ip 00007fee37a417e8 sp 00007ffd09a4aaf0 error 4 in ld-2.17.so[7fee37a36000+22000]
LustreError: 20460:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x18c8:0x0]: rc = -5
LustreError: 20460:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
17[20553]: segfault at 0 ip 0000000000403e5f sp 00007ffc41e03780 error 6 in 17[400000+6000]
LustreError: 9993:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0x97e:0x0]: rc = -2
LustreError: 15914:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e77b92a8: inode [0x200000405:0x97e:0x0] mdc close failed: rc = -2
LustreError: 15914:0:(file.c:248:ll_close_inode_openhandle()) Skipped 6 previous similar messages
LustreError: 22497:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x18c8:0x0]: rc = -5
LustreError: 22497:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
3[24661]: segfault at 8 ip 00007f8bb3abb7e8 sp 00007ffc60619930 error 4 in ld-2.17.so[7f8bb3ab0000+22000]
17[26257]: segfault at 8 ip 00007f3b268d57e8 sp 00007ffdb718e0b0 error 4 in ld-2.17.so[7f3b268ca000+22000]
LustreError: 26319:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x18c8:0x0]: rc = -5
LustreError: 26319:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 21519:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 21519:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 357 previous similar messages
Lustre: 21519:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 21519:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 357 previous similar messages
Lustre: 21519:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 21519:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 357 previous similar messages
Lustre: 21519:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 21519:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 357 previous similar messages
Lustre: 21519:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21519:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 357 previous similar messages
Lustre: 22657:0:(mdd_dir.c:4826:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/12 is open, migrate only dentry
Lustre: 22657:0:(mdd_dir.c:4826:mdd_migrate_object()) Skipped 5 previous similar messages
14[2866]: segfault at 8 ip 00007f9b1041c7e8 sp 00007ffd85501060 error 4 in ld-2.17.so[7f9b10411000+22000]
1[3666]: segfault at 8 ip 00007fd76affd7e8 sp 00007ffcab0911d0 error 4 in ld-2.17.so[7fd76aff2000+22000]
8[4074]: segfault at 8 ip 00007fb55ee0c7e8 sp 00007fff952fc930 error 4 in ld-2.17.so[7fb55ee01000+22000]
10[4938]: segfault at 8 ip 00007fcbe957e7e8 sp 00007ffd993145c0 error 4 in ld-2.17.so[7fcbe9573000+22000]
5[5408]: segfault at 8 ip 00007f13162597e8 sp 00007fffde4bcef0 error 4 in ld-2.17.so[7f131624e000+22000]
1[5216]: segfault at 8 ip 00007f0d5720c7e8 sp 00007ffd2f71df50 error 4 in ld-2.17.so[7f0d57201000+22000]
LustreError: 22657:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000404:0x1b29:0x0]/12 failed: rc = -2
LustreError: 7656:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x200000403:0x3154:0x0] dir layout mismatch:
LustreError: 7656:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 7656:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0x37:0x0]
LustreError: 7656:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 9003:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x1a47:0x0]: rc = -5
LustreError: 7659:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=3 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 7659:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 9003:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 32001:0:(out_handler.c:879:out_tx_end()) lustre-MDT0001-osd: error during execution of #0 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:141: rc = -2
3[11159]: segfault at 8 ip 00007fe0c46957e8 sp 00007ffca3ff8ea0 error 4 in ld-2.17.so[7fe0c468a000+22000]
LustreError: 22657:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/12 failed: rc = -2
Lustre: dir [0x240000404:0x1be0:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000404:0x19e8:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 14685:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x1ac9:0x0]: rc = -5
LustreError: 14685:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 14685:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 14685:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 1 previous similar message
Lustre: dir [0x240000403:0x1a8a:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 22389:0:(mdd_dir.c:4747:mdd_migrate_cmd_check()) lustre-MDD0002: '4' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 4' to finish migration: rc = -1
Lustre: dir [0x280000403:0x1919:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 22389:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000405:0xe08:0x0]/16 failed: rc = -2
LustreError: 22389:0:(mdt_reint.c:2564:mdt_reint_migrate()) Skipped 4 previous similar messages
Lustre: 21140:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x1050:0x0] with magic=0xbd60bd0
Lustre: 21140:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 22500:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x1be0:0x0]: rc = -5
LustreError: 22500:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 18838:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x1919:0x0]: rc = -2
LustreError: 21810:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x280000403:0x1919:0x0] mdc close failed: rc = -2
Lustre: 22139:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x1cb0:0x0] with magic=0xbd60bd0
Lustre: 22139:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 11 previous similar messages
LustreError: 28448:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x1c69:0x0]: rc = -5
LustreError: 28448:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 5 previous similar messages
LustreError: 28448:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 28448:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 5 previous similar messages
Lustre: 11845:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 618, rollback = 7
Lustre: 11845:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 191 previous similar messages
19[29149]: segfault at 8 ip 00007fbee09f57e8 sp 00007ffc1be06490 error 4 in ld-2.17.so[7fbee09ea000+22000]
LustreError: 17503:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x1919:0x0]: rc = -2
Lustre: 9988:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x1daa:0x0] with magic=0xbd60bd0
Lustre: 9988:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 22428:0:(mdt_reint.c:2564:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/17 failed: rc = -16
1[9520]: segfault at 8 ip 00007f92cdbe37e8 sp 00007fff87d6cb10 error 4 in ld-2.17.so[7f92cdbd8000+22000]
14[14226]: segfault at 8 ip 00007fb1810a67e8 sp 00007ffecfa04d50 error 4 in ld-2.17.so[7fb18109b000+22000]
12[14471]: segfault at 8 ip 00007f9f51a8c7e8 sp 00007ffe2d039e90 error 4 in ld-2.17.so[7f9f51a81000+22000]
12[14543]: segfault at 8 ip 00007fb082e9b7e8 sp 00007ffd7cf0ef00 error 4 in ld-2.17.so[7fb082e90000+22000]
10[14685]: segfault at 8 ip 00007f12e2aca7e8 sp 00007ffd8c8bdaa0 error 4 in ld-2.17.so[7f12e2abf000+22000]
6[15323]: segfault at 8 ip 00007f6b72e107e8 sp 00007ffe6cb87a70 error 4 in ld-2.17.so[7f6b72e05000+22000]
17[20562]: segfault at 401ac0 ip 0000000000401ac0 sp 00007ffecc36c698 error 7 in 17[400000+6000]
17[20913]: segfault at 8 ip 00007fef5c6067e8 sp 00007ffe5323f130 error 4 in ld-2.17.so[7fef5c5fb000+22000]
11[21949]: segfault at 406000 ip 0000000000406000 sp 00007fffad969318 error 14 in 11[606000+1000]
15[24905]: segfault at 8 ip 00007f13e0ec47e8 sp 00007ffcd871cb10 error 4 in ld-2.17.so[7f13e0eb9000+22000]
10[25481]: segfault at 0 ip 0000000000403e5f sp 00007ffc2c27ee20 error 6 in 10[400000+6000]
LustreError: 21053:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e858dd28: inode [0x200000403:0x3a46:0x0] mdc close failed: rc = -13
LustreError: 21053:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 15975:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x2067:0x0] with magic=0xbd60bd0
Lustre: 15975:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 3 previous similar messages
18[3683]: segfault at 8 ip 00007f11423657e8 sp 00007ffec8bf0c00 error 4 in ld-2.17.so[7f114235a000+22000]
Lustre: 11214:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 498 < left 1055, rollback = 2
Lustre: 11214:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 729 previous similar messages
8[9910]: segfault at 406000 ip 0000000000405fff sp 00007ffd6b83dd70 error 14 in 8[400000+6000]
8[9309]: segfault at 4045bc ip 00000000004045bc sp 00007ffcb5de50a8 error 7 in 8[400000+6000]
8[12751]: segfault at 8 ip 00007fb4a5ee07e8 sp 00007fff029dae70 error 4 in ld-2.17.so[7fb4a5ed5000+22000]
18[12945]: segfault at 1c ip 00000000004017b0 sp 00007ffd8c0bac90 error 6 in 18 (deleted)[400000+6000]
6[16121]: segfault at 8 ip 00007f7cc08197e8 sp 00007ffdc9f95590 error 4 in ld-2.17.so[7f7cc080e000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2e156d067 PUD 2c7c82067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata
CPU: 2 PID: 28975 Comm: ll_sa_28963 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802da325c40 ti: ffff8802ce3dc000 task.ti: ffff8802ce3dc000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802ce3dfd78 EFLAGS: 00010296
RAX: ffff8802c50bf310 RBX: 0000000000000008 RCX: 000000010026001c
RDX: 0000000000000026 RSI: ffff8802c50bf640 RDI: 0000000000000008
RBP: ffff8802ce3dfd88 R08: ffff8802c8ef6368 R09: 0000000000000000
R10: ffff8802c8ef0d48 R11: ffff8802c8ef6368 R12: 0000000000000000
R13: ffff8802cbb91280 R14: ffff8802c50bf640 R15: ffff8802c8ef6368
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000a05fa000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1110601>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa110f850>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

dd (15831) used greatest stack depth: 9600 bytes left
LustreError: 15933:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0054a88: inode [0x200000401:0x31:0x0] mdc close failed: rc = -13
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
LustreError: 16231:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0054a88: inode [0x200000401:0x81:0x0] mdc close failed: rc = -13
Lustre: 10431:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a97e3c40 x1834598606298368/t4294970122(0) o101->8960cd2f-085c-4aa8-9f8c-16c929fc9fe2@0@lo:589/0 lens 376/816 e 0 to 0 dl 1749609654 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 16202:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 16202:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 16202:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16202:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16202:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 16202:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16202:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 16202:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16202:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16202:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16202:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16202:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 618, rollback = 7
Lustre: 10395:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10395:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10395:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10395:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10395:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10395:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
4[21550]: segfault at 406000 ip 0000000000406000 sp 00007ffd497dbdd8 error 14 in 1[606000+1000]
1[21558]: segfault at 406000 ip 0000000000406000 sp 00007fff7061b068 error 14 in 1[606000+1000]
Lustre: 19370:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 19370:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 19370:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 19370:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 19370:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 19370:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 19370:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 19370:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 19370:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 19370:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 19370:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 19370:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10397:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10397:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 10397:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10397:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10397:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10397:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10397:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10397:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10397:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10397:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10397:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10397:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 14656:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x401:0x0] with magic=0xbd60bd0
12[29067]: segfault at 8 ip 00007f3e52d417e8 sp 00007ffcdea7b730 error 4 in ld-2.17.so[7f3e52d36000+22000]
LustreError: 29597:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee98b7e8: inode [0x200000402:0x3ee:0x0] mdc close failed: rc = -13
Lustre: 15105:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15105:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 13 previous similar messages
Lustre: 15105:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15105:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 15105:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15105:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 15105:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 15105:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 15105:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15105:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 15105:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15105:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 13 previous similar messages
15[1413]: segfault at 8 ip 00007fbeed3807e8 sp 00007fff899201d0 error 4 in ld-2.17.so[7fbeed375000+22000]
LustreError: 1413:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee98b7e8: inode [0x200000401:0x577:0x0] mdc close failed: rc = -13
LustreError: 4101:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee98b7e8: inode [0x200000402:0x69f:0x0] mdc close failed: rc = -13
Lustre: 14437:0:(lod_lov.c:1402:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x638:0x0] with magic=0xbd60bd0
Lustre: 14437:0:(lod_lov.c:1402:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 6647:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X0.
10[8692]: segfault at 8 ip 00007fbca47287e8 sp 00007fffbd6b89c0 error 4 in ld-2.17.so[7fbca471d000+22000]
16[9865]: segfault at 0 ip 0000000000403e5f sp 00007fff46b7b7f0 error 6 in 16[400000+6000]
Lustre: 15105:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15105:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 15105:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15105:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 15105:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15105:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 15105:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15105:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 15105:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15105:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 15105:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15105:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 17 previous similar messages
17[10597]: segfault at 8 ip 00007fe0ccdf07e8 sp 00007ffc6c287b80 error 4 in ld-2.17.so[7fe0ccde5000+22000]
LustreError: 10597:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee98b7e8: inode [0x200000401:0x862:0x0] mdc close failed: rc = -13
1[16991]: segfault at 8 ip 00007f68cd6a27e8 sp 00007ffdc09eafa0 error 4 in ld-2.17.so[7f68cd697000+22000]
6[22532]: segfault at 8 ip 00007f8e86df67e8 sp 00007ffdfb5cc060 error 4 in ld-2.17.so[7f8e86deb000+22000]
LustreError: 22532:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0054a88: inode [0x200000402:0xb9e:0x0] mdc close failed: rc = -13

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x300000402 to 0x300000405
PGD 2dd918067 PUD 2da720067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) llz4hc(OE) llz4(OE) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix virtio_blk i2c_core serio_raw libata floppy
CPU: 11 PID: 22641 Comm: ll_sa_22487 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88023f0c3760 ti: ffff880268608000 task.ti: ffff880268608000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88026860bd78 EFLAGS: 00010296
RAX: ffff88022d111410 RBX: 0000000000000008 RCX: 0000000100260021
RDX: 0000000000000026 RSI: ffff88022d111740 RDI: 0000000000000008
RBP: ffff88026860bd88 R08: ffff880249eac248 R09: 0000000000000000
R10: ffff880249eab1b8 R11: ffff880249eac248 R12: 0000000000000000
R13: ffff88023f0c49d0 R14: ffff88022d111740 R15: ffff880249eac248
FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002e0aa2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa110dbc1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa110ce10>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 27278:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c8e10540 x1833852604476672/t4295172078(0) o101->1ac643b5-50aa-4fc6-833e-44e40151ac16@0@lo:736/0 lens 376/18608 e 0 to 0 dl 1748900856 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 27278:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0002: opcode 0: before 515 < left 2897, rollback = 0
Lustre: 27278:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1611 previous similar messages
Lustre: 27278:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 6/24/0, destroy: 1/4/0
Lustre: 27278:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 2210 previous similar messages
Lustre: 27278:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 2897/2897/0, xattr_set: 4345/40664/0
Lustre: 27278:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 2210 previous similar messages
Lustre: 27278:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 28/157/0, punch: 0/0/0, quota 1/3/0
Lustre: 27278:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 2210 previous similar messages
Lustre: 27278:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 7/118/0, delete: 2/5/0
Lustre: 27278:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 2210 previous similar messages
Lustre: 27278:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0
Lustre: 27278:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 2210 previous similar messages
Lustre: 31870:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88027d2bee40 x1833852604921088/t4295252671(0) o101->1ac643b5-50aa-4fc6-833e-44e40151ac16@0@lo:737/0 lens 376/33392 e 0 to 0 dl 1748900857 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 27278:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802aa9e7840 x1833852605998080/t4295172393(0) o101->1ac643b5-50aa-4fc6-833e-44e40151ac16@0@lo:741/0 lens 376/19432 e 0 to 0 dl 1748900861 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10891:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802aba0b240 x1833852607630336/t4295253003(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:744/0 lens 376/33392 e 0 to 0 dl 1748900864 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10886:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e14d1440 x1833852610529920/t4295172715(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:754/0 lens 376/45904 e 0 to 0 dl 1748900874 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10886:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
LustreError: 5018:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000409:0x2a36:0x0]: rc = -5
LustreError: 5018:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 62 previous similar messages
LustreError: 5018:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 5018:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 62 previous similar messages
Lustre: 23495:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88025f9dee40 x1833852612979072/t4295173015(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:9/0 lens 376/45904 e 0 to 0 dl 1748900884 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 23495:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
Lustre: 17272:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000406:0x2bb1:0x0] with magic=0xbd60bd0
Lustre: 17272:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 137 previous similar messages
LustreError: 394:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 10 [0x240000406:0x2bb7:0x0] inode@0000000000000000: rc = -5
LustreError: 394:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 6 previous similar messages
Lustre: 20454:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802de479940 x1833852622441472/t4295173703(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:29/0 lens 376/45904 e 0 to 0 dl 1748900904 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 20454:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 3 previous similar messages
Lustre: 27291:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d07c3740 x1833852643091072/t4295256115(0) o101->1ac643b5-50aa-4fc6-833e-44e40151ac16@0@lo:86/0 lens 376/43216 e 0 to 0 dl 1748900961 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 27291:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff8802d1d9f0c0/0x235b843c035bb25 lrc: 3/0,0 mode: PW/PW res: [0x340000402:0xbb3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->786431) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x235b843c035b3f4 expref: 3025 pid: 4236 timeout: 3403 lvb_type: 0
LustreError: 4225:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a775ca88 ns: filter-lustre-OST0002_UUID lock: ffff880293cee940/0x235b843c04d0c11 lrc: 3/0,0 mode: --/PW res: [0x340000402:0xca4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x235b843c04d0c03 expref: 2994 pid: 4225 timeout: 0 lvb_type: 0
LustreError: 4225:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 8 previous similar messages
LustreError: lustre-OST0002-osc-ffff8802e56392a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff8802e56392a8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff8802e56392a8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 12679:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88024eaf8040 x1833852650850816/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: 3799:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000406:0x1c3d:0x0]/ may get corrupted (rc -108)
Lustre: 3797:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000409:0x2c5d:0x0]// may get corrupted (rc -108)
LustreError: 13104:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802850bbc40 x1833852651299328/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 13104:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: 13337:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0002-osc-ffff8802e56392a8: namespace resource [0x340000402:0x160f:0x0].0x0 (ffff8802682efc40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffff8802e56392a8: Connection restored to (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 10672:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007aeda840 x1833852655982208/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0003-osc-ffff8802e45db7e8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: lustre-OST0000-osc-ffff8802e45db7e8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3794:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000403:0x50b2:0x0]/ may get corrupted (rc -108)
LustreError: 14289:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0000-osc-ffff8802e45db7e8: namespace resource [0x2c0000401:0x20ef:0x0].0x0 (ffff88025450b9c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 9157:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802caaacb40 x1833852657175680/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 9157:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 36 previous similar messages
Lustre: 19203:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880293f28040 x1833852676968064/t4295177425(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:213/0 lens 376/46696 e 0 to 0 dl 1748901088 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 19203:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 4 previous similar messages
ptlrpc_watchdog_fire: 2 callbacks suppressed
Lustre: mdt_io00_009: service thread pid 19718 was inactive for 40.096 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 19718, comm: mdt_io00_009 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_lock_two_dirs+0x30/0x190 [mdt]
[<0>] mdt_reint_rename+0x1192/0x3950 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880241184780/0x235b843c05ae4aa lrc: 3/0,0 mode: PW/PW res: [0x380000402:0x23dd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x235b843c05adce6 expref: 3977 pid: 9796 timeout: 3558 lvb_type: 0
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 10016:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d2392548 ns: filter-lustre-OST0003_UUID lock: ffff8802378661c0/0x235b843c071f4cf lrc: 3/0,0 mode: --/PW res: [0x380000401:0x3bf4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x235b843c071f4c8 expref: 3730 pid: 10016 timeout: 0 lvb_type: 0
LustreError: 9230:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802aa9e2d40 x1833852718928768/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 9230:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 16 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e56392a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 196 previous similar messages
Lustre: lustre-OST0003-osc-ffff8802e56392a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e56392a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 22959:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802e56392a8: namespace resource [0x380000402:0x33ce:0x0].0x0 (ffff880258d6efc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 22959:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1516 previous similar messages
Lustre: lustre-OST0003-osc-ffff8802e56392a8: Connection restored to (at 0@lo)
Lustre: Skipped 2 previous similar messages
Lustre: 19696:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1748901077/real 1748901077] req@ffff88025574c140 x1833852695645056/t0(0) o101->lustre-MDT0001-mdc-ffff8802e56392a8@0@lo:12/10 lens 376/14824 e 0 to 1 dl 1748901134 ref 2 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0
Lustre: lustre-MDT0001: Client d96ae9f5-416f-4987-9648-bf5c9c55fb45 (at 0@lo) reconnecting
Lustre: mdt_io00_009: service thread pid 19718 completed after 53.120s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 2994:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0001: [0x240000406:0x3027:0x0] doesn't exist!: rc = -14
Lustre: 20401:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880257b71440 x1833852745298944/t4295185973(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:422/0 lens 376/48256 e 0 to 0 dl 1748901297 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 20401:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 12 previous similar messages
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff8802354e4f00/0x235b843c073c6aa lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0x3622:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0x235b843c073c6a3 expref: 2888 pid: 14244 timeout: 3666 lvb_type: 0
LustreError: 9219:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e45d8008 ns: filter-lustre-OST0000_UUID lock: ffff880252dcb880/0x235b843c0893488 lrc: 3/0,0 mode: --/PW res: [0x2c0000402:0x358a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000080020000 nid: 0@lo remote: 0x235b843c0893290 expref: 2888 pid: 9219 timeout: 0 lvb_type: 0
LustreError: lustre-OST0000-osc-ffff8802e56392a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: 17495:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b324d040 x1833852765000960/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 17495:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 14 previous similar messages
LustreError: lustre-OST0000-osc-ffff8802e56392a8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3803:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x240000406:0x31a7:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff8802e56392a8: Connection restored to (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff8802e45db7e8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: 9977:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88026f51b240 x1833852774429568/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 9977:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e45db7e8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3806:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000403:0x5395:0x0]/ may get corrupted (rc -108)
Lustre: 3806:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000403:0x538e:0x0]/ may get corrupted (rc -108)
Lustre: 3808:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000406:0x205e:0x0]/ may get corrupted (rc -108)
LustreError: 31204:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802e45db7e8: namespace resource [0x380000402:0x375e:0x0].0x0 (ffff880258ae5440) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31204:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 271 previous similar messages
Lustre: mdt_out00_004: service thread pid 20833 was inactive for 40.056 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 20833, comm: mdt_out00_004 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1844/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 20833:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a6d8b7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff880259e3c3c0/0x235b843c094e3c6 lrc: 3/0,0 mode: PR/PR res: [0x200000407:0x5749:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x235b843c08629d6 expref: 52 pid: 20833 timeout: 0 lvb_type: 0
Lustre: mdt_out00_004: service thread pid 20833 completed after 70.335s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802e56392a8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 32488:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e56392a8: inode [0x200000409:0x2d59:0x0] mdc close failed: rc = -108
LustreError: 32488:0:(file.c:247:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 30335:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 30335:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 36 previous similar messages
Lustre: lustre-OST0003-osc-MDT0001: update sequence from 0x380000401 to 0x380000403
Lustre: ll_ost00_019: service thread pid 11540 was inactive for 40.073 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 11540, comm: ll_ost00_019 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x19c/0x610 [ofd]
[<0>] ofd_destroy_hdl+0x20c/0xaf0 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost00_206: service thread pid 14217 was inactive for 40.078 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: ll_ost00_058: service thread pid 6293 was inactive for 39.990 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 5 previous similar messages
LustreError: 21482:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880320065540 x1833852798345984/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 21482:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: 8722:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748901347 with bad export cookie 159235962974058847
LustreError: lustre-OST0000-osc-ffff8802e56392a8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: 8722:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages
Lustre: 3804:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x280000406:0x213b:0x0]/ may get corrupted (rc -108)
Lustre: ll_ost00_019: service thread pid 11540 completed after 46.165s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_052: service thread pid 5577 completed after 46.141s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_058: service thread pid 6293 completed after 45.457s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_109: service thread pid 9160 completed after 46.258s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_206: service thread pid 14217 completed after 46.259s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_191: service thread pid 13100 completed after 46.234s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_056: service thread pid 5658 completed after 46.239s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_001: service thread pid 13293 completed after 46.340s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88023e10f840/0x235b843c0914764 lrc: 3/0,0 mode: PW/PW res: [0x340000401:0x6bd5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000480000020 nid: 0@lo remote: 0x235b843c091475d expref: 4374 pid: 14217 timeout: 3811 lvb_type: 0
LustreError: 10675:0:(ldlm_lockd.c:257:expired_lock_main()) Skipped 3 previous similar messages
LustreError: lustre-OST0002-osc-ffff8802e45db7e8: operation ost_sync to node 0@lo failed: rc = -107
LustreError: Skipped 7 previous similar messages
LustreError: 7946:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748901379 with bad export cookie 159235962968368904
LustreError: lustre-OST0002-osc-ffff8802e45db7e8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 14231:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ca8defc8 ns: filter-lustre-OST0002_UUID lock: ffff88024b1f9a80/0x235b843c0a0d227 lrc: 3/0,0 mode: --/PW res: [0x340000400:0x513f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x235b843c0a0d220 expref: 3789 pid: 14231 timeout: 0 lvb_type: 0
LustreError: 4094:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0002-osc-ffff8802e45db7e8: namespace resource [0x340000400:0x4ea0:0x0].0x0 (ffff88008d3387c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 4094:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 43 previous similar messages
Lustre: lustre-OST0002: haven't heard from client 1ac643b5-50aa-4fc6-833e-44e40151ac16 (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp ffff880294ffefc8, cur 1748901410 deadline 1748901409 last 1748901379
Lustre: lustre-OST0001-osc-MDT0001: update sequence from 0x300000401 to 0x300000403
Lustre: lustre-OST0002-osc-MDT0001: update sequence from 0x340000401 to 0x340000403
Lustre: lustre-OST0002-osc-ffff8802e45db7e8: Connection restored to (at 0@lo)
Lustre: Skipped 3 previous similar messages
LustreError: lustre-OST0002-osc-ffff8802e45db7e8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 6785:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0002-osc-ffff8802e45db7e8: namespace resource [0x340000400:0x5fd8:0x0].0x0 (ffff880078470040) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 6785:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 197 previous similar messages
Lustre: lustre-OST0001-osc-MDT0002: update sequence from 0x300000400 to 0x300000404
Lustre: 18329:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 0: before 513 < left 1897, rollback = 0
Lustre: 18329:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1473 previous similar messages
Lustre: 18329:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 6/24/0, destroy: 1/4/0
Lustre: 18329:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1473 previous similar messages
Lustre: 18329:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1897/1897/0, xattr_set: 2845/26664/0
Lustre: 18329:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1473 previous similar messages
Lustre: 18329:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 28/157/0, punch: 0/0/0, quota 1/3/0
Lustre: 18329:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1473 previous similar messages
Lustre: 18329:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 7/118/0, delete: 2/5/1
Lustre: 18329:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1473 previous similar messages
Lustre: 18329:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 18329:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1467 previous similar messages
Lustre: 20320:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c8e12840 x1833852831823488/t4295192286(0) o101->d96ae9f5-416f-4987-9648-bf5c9c55fb45@0@lo:682/0 lens 376/48256 e 0 to 0 dl 1748901557 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 20320:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 14 previous similar messages
Lustre: lustre-OST0000-osc-MDT0002: update sequence from 0x2c0000400 to 0x2c0000403
Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x2c0000401 to 0x2c0000404
Lustre: lustre-OST0003-osc-MDT0002: update sequence from 0x380000400 to 0x380000404
Lustre: lustre-OST0003-osc-ffff8802e56392a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 4 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e56392a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 11540:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802e9d98f40 x1833852865741696/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 11540:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 39 previous similar messages
LustreError: 9215:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e40a6fc8 ns: filter-lustre-OST0003_UUID lock: ffff8800750d8f40/0x235b843c0c21488 lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x7c44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x235b843c0c21449 expref: 5213 pid: 9215 timeout: 0 lvb_type: 0
Lustre: 3806:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x240000408:0x1f10:0x0]/ may get corrupted (rc -108)
Lustre: 3802:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x20000040a:0x1e3:0x0]/ may get corrupted (rc -108)
Lustre: 3804:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x240000408:0x1efd:0x0]// may get corrupted (rc -108)
Lustre: 3804:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x240000408:0x1ef0:0x0]// may get corrupted (rc -108)
LustreError: 14147:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802e56392a8: namespace resource [0x380000400:0x6b30:0x0].0x0 (ffff8802d1dd9bc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 14147:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message
hrtimer: interrupt took 22644563 ns
Lustre: lustre-OST0003: haven't heard from client d96ae9f5-416f-4987-9648-bf5c9c55fb45 (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp ffff88009c29e678, cur 1748901588 deadline 1748901587 last 1748901557
Lustre: lustre-OST0002-osc-MDT0002: update sequence from 0x340000400 to 0x340000404
LustreError: 14147:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802e56392a8: namespace resource [0x380000403:0x270d:0x0].0x0 (ffff8802a7d7c2c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 14147:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 53 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e56392a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 22 previous similar messages
LustreError: lustre-OST0003-osc-ffff8802e56392a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2db2a0067 PUD 2d08fd067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) llz4hc(OE) llz4(OE) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks i2c_core virtio_blk ata_piix serio_raw floppy libata
CPU: 2 PID: 2039 Comm: ll_sa_1915 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800705ea4f0 ti: ffff8800655e0000 task.ti: ffff8800655e0000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8800655e3d78 EFLAGS: 00010296
RAX: ffff880218da1d90 RBX: 0000000000000008 RCX: dead000000000200
RDX: 0000000000000002 RSI: ffff880218da20c0 RDI: 0000000000000008
RBP: ffff8800655e3d88 R08: ffff880331403660 R09: 0000000000000000
R10: ffff880331403640 R11: ffff880254aa93e8 R12: 0000000000000000
R13: ffff88025bce0010 R14: ffff880218da20c0 R15: ffff880254aa93e8
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000027d818000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa112abc1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa1129e10>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 19422:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88031fef5a40 x1833413651556992/t4295160221(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:25/0 lens 376/16544 e 0 to 0 dl 1748481875 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 17697:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88028163f840 x1833413651916032/t4295317188(0) o101->49c1553c-0fc9-4b6e-a0af-6e671e257aff@0@lo:26/0 lens 376/14192 e 0 to 0 dl 1748481876 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 23753:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x4bea:0x0] doesn't exist!: rc = -14
Lustre: 28960:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880290c76940 x1833413652345344/t4295161986(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:27/0 lens 376/21640 e 0 to 0 dl 1748481877 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 28960:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
Lustre: 11000:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008bc74b40 x1833413652832256/t4295162104(0) o101->49c1553c-0fc9-4b6e-a0af-6e671e257aff@0@lo:30/0 lens 376/34184 e 0 to 0 dl 1748481880 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11281:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880290c75540 x1833413658893824/t4295161139(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:53/0 lens 376/43360 e 0 to 0 dl 1748481903 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10990:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008d6d0a40 x1833413662171776/t4295161277(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:61/0 lens 376/43360 e 0 to 0 dl 1748481911 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11281:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008bc75f40 x1833413681911936/t4295320744(0) o101->49c1553c-0fc9-4b6e-a0af-6e671e257aff@0@lo:100/0 lens 376/48184 e 0 to 0 dl 1748481950 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11281:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages
LustreError: 29575:0:(statahead.c:2447:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: 11964:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880261cd0540 x1833413705501056/t4295167535(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:141/0 lens 376/44848 e 0 to 0 dl 1748481991 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11964:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 4 previous similar messages
LustreError: lustre-MDT0001-mdc-ffff8800a72fb7e8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 20681:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802591e0a40 x1833413743217536/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 20681:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 13207:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e5042548 ns: filter-lustre-OST0003_UUID lock: ffff8802689eb100/0xf43bd31648b5876a lrc: 3/0,0 mode: --/PW res: [0x340000402:0x2e9e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0xf43bd31648b584ed expref: 4056 pid: 13207 timeout: 0 lvb_type: 0
LustreError: lustre-OST0003-osc-ffff8800a72fb7e8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 25491:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880234b69940 x1833413743301632/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 25491:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
LustreError: lustre-OST0000-osc-ffff8800a72fb7e8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3961:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xc4:0x0]// may get corrupted (rc -108)
Lustre: 3959:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xce:0x0]// may get corrupted (rc -108)
Lustre: 3965:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x240000404:0x4dd1:0x0]/ may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000406:0x1a4e:0x0]// may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000406:0x1a2a:0x0]/ may get corrupted (rc -108)
LustreError: 3454:0:(ldlm_resource.c:1097:ldlm_resource_complain()) lustre-OST0003-osc-ffff8800a72fb7e8: namespace resource [0x340000402:0x348e:0x0].0x0 (ffff88006be82d40) refcount nonzero (0) after lock cleanup; forcing cleanup.
LustreError: 3454:0:(ldlm_resource.c:1097:ldlm_resource_complain()) Skipped 1 previous similar message
LustreError: 3472:0:(ldlm_resource.c:1097:ldlm_resource_complain()) lustre-OST0000-osc-ffff8800a72fb7e8: namespace resource [0x2c0000401:0x310f:0x0].0x0 (ffff880248fd3c40) refcount nonzero (0) after lock cleanup; forcing cleanup.
LustreError: 3472:0:(ldlm_resource.c:1097:ldlm_resource_complain()) Skipped 467 previous similar messages
Lustre: 18246:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88006aa2df40 x1833413745646080/t4295324283(0) o101->49c1553c-0fc9-4b6e-a0af-6e671e257aff@0@lo:233/0 lens 376/48488 e 0 to 0 dl 1748482083 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 18246:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 8 previous similar messages
LustreError: 8221:0:(statahead.c:2447:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: 28933:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 6/24/0, destroy: 1/4/0
Lustre: 28933:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 2640 previous similar messages
Lustre: 28933:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1977/1977/0, xattr_set: 2965/27784/0
Lustre: 28933:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 2640 previous similar messages
Lustre: 28933:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 28/157/0, punch: 0/0/0, quota 1/3/0
Lustre: 28933:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 2640 previous similar messages
Lustre: 28933:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 7/118/0, delete: 2/5/1
Lustre: 28933:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 2640 previous similar messages
Lustre: 28933:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 28933:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 2640 previous similar messages
Lustre: 11282:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88023b941940 x1833413808336512/t4295170346(0) o101->ebe878d2-f553-43ee-90ff-4d334c9ddb8a@0@lo:388/0 lens 376/43360 e 0 to 0 dl 1748482238 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11282:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 6 previous similar messages
Lustre: 16951:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0001: opcode 0: before 514 < left 3515, rollback = 0
Lustre: 16951:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1442 previous similar messages
LustreError: lustre-MDT0000-mdc-ffff8802e5748008: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 12471:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 8201:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x2:0x0] error: rc = -108
LustreError: lustre-MDT0002-mdc-ffff8800a72fb7e8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 23753:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88028f50ae98 ns: mdt-lustre-MDT0002_UUID lock: ffff88026870d680/0xf43bd31648c825ec lrc: 3/0,0 mode: PR/PR res: [0x200000409:0xfa8:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xf43bd31648c825d0 expref: 2 pid: 23753 timeout: 0 lvb_type: 0
Lustre: lustre-OST0002-osc-MDT0002: update sequence from 0x380000400 to 0x380000403
Lustre: ll_ost00_198: service thread pid 26857 was inactive for 72.058 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 26857, comm: ll_ost00_198 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x19c/0x610 [ofd]
[<0>] ofd_destroy_hdl+0x20c/0xaf0 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
Lustre: ll_ost00_020: service thread pid 2100 was inactive for 72.060 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
[<0>] 0xfffffffffffffffe
Pid: 29651, comm: ll_ost00_219 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x19c/0x610 [ofd]
[<0>] ofd_destroy_hdl+0x20c/0xaf0 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22413, comm: ll_ost00_070 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x19c/0x610 [ofd]
[<0>] ofd_destroy_hdl+0x20c/0xaf0 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost00_034: service thread pid 21175 was inactive for 72.112 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0003-osc-MDT0002: update sequence from 0x340000400 to 0x340000403
Lustre: 23683:0:(out_handler.c:879:out_tx_end()) lustre-MDT0001-osd: error during execution of #2 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:562: rc = -2
LustreError: 23683:0:(out_lib.c:1168:out_tx_index_delete_undo()) lustre-MDT0001-osd: Oops, can not rollback index_delete yet: rc = -524
LustreError: 23683:0:(out_lib.c:1168:out_tx_index_delete_undo()) Skipped 3 previous similar messages
Lustre: ll_ost00_020: service thread pid 2100 completed after 89.269s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_127: service thread pid 24013 completed after 88.441s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_113: service thread pid 23940 completed after 88.438s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_198: service thread pid 26857 completed after 89.298s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_126: service thread pid 24007 completed after 88.455s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_070: service thread pid 22413 completed after 89.423s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_012: service thread pid 24209 completed after 88.492s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_155: service thread pid 24694 completed after 87.812s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_174: service thread pid 24764 completed after 88.492s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_091: service thread pid 23116 completed after 88.945s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_094: service thread pid 23133 completed after 87.834s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_066: service thread pid 22352 completed after 87.799s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_044: service thread pid 21271 completed after 87.806s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_193: service thread pid 26473 completed after 87.802s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_144: service thread pid 24676 completed after 87.842s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_122: service thread pid 23985 completed after 87.831s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_035: service thread pid 21176 completed after 87.812s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_166: service thread pid 24739 completed after 87.810s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_219: service thread pid 29651 completed after 89.331s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_206: service thread pid 28478 completed after 89.338s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_014: service thread pid 24876 completed after 89.334s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_202: service thread pid 28468 completed after 87.843s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_129: service thread pid 24043 completed after 87.859s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_036: service thread pid 21183 completed after 89.400s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_199: service thread pid 27101 completed after 88.569s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_171: service thread pid 24761 completed after 87.876s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_135: service thread pid 24655 completed after 89.456s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_027: service thread pid 20746 completed after 89.571s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_142: service thread pid 24666 completed after 87.962s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_053: service thread pid 22091 completed after 87.960s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_034: service thread pid 21175 completed after 88.776s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_002: service thread pid 13209 completed after 88.650s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 10932:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 11281:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
Lustre: lustre-OST0000-osc-MDT0002: update sequence from 0x2c0000400 to 0x2c0000403
Lustre: lustre-OST0001-osc-MDT0002: update sequence from 0x300000400 to 0x300000403
LustreError: 11961:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 11961:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 2 previous similar messages
Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x2c0000401 to 0x2c0000404
LustreError: 26938:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
Lustre: lustre-OST0002-osc-MDT0001: update sequence from 0x380000401 to 0x380000404
Lustre: lustre-OST0003-osc-MDT0001: update sequence from 0x340000401 to 0x340000404
Lustre: lustre-OST0001-osc-MDT0001: update sequence from 0x300000401 to 0x300000404
Lustre: ll_ost_out00_00: service thread pid 13221 was inactive for 40.154 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 21 previous similar messages
Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000405
Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x380000402 to 0x380000405
LustreError: 10885:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8802687c6940/0xf43bd3164908ec83 lrc: 3/0,0 mode: PW/PW res: [0x340000402:0x506f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000480020020 nid: 0@lo remote: 0xf43bd3164908ec75 expref: 5148 pid: 24209 timeout: 3458 lvb_type: 0
LustreError: 10885:0:(ldlm_lockd.c:257:expired_lock_main()) Skipped 6 previous similar messages
LustreError: 17427:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748482371 with bad export cookie 17598892061542943214
LustreError: 24687:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88021fcaad40 x1833413935935360/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 24687:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 9 previous similar messages
Lustre: lustre-OST0003-osc-ffff8800a72fb7e8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff8800a72fb7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 471 previous similar messages
Lustre: Skipped 6 previous similar messages
LustreError: lustre-OST0003-osc-ffff8800a72fb7e8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3965:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000409:0x1594:0x0]/ may get corrupted (rc -108)
Lustre: 3965:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000409:0x15b0:0x0]/ may get corrupted (rc -108)
Lustre: 3965:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000407:0x19ee:0x0]/ may get corrupted (rc -108)
Lustre: 3952:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x4d6:0x0]/ may get corrupted (rc -108)
Lustre: 3959:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000409:0x15c6:0x0]// may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x459:0x0]/ may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x20000040a:0x46a:0x0]/ may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x40b:0x0]/ may get corrupted (rc -108)
LustreError: 2100:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880230f1e440 x1833413936458368/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 2100:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 33 previous similar messages
Lustre: lustre-OST0003-osc-ffff8800a72fb7e8: Connection restored to 192.168.123.12@tcp (at 0@lo)
Lustre: Skipped 6 previous similar messages
Lustre: 23753:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff88008c9e3240 x1833413937464832/t4295338584(0) o101->49c1553c-0fc9-4b6e-a0af-6e671e257aff@0@lo:658/0 lens 376/48488 e 0 to 0 dl 1748482508 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 23753:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 24 previous similar messages
Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x300000402 to 0x300000405
Lustre: lustre-OST0002-osc-MDT0002: update sequence from 0x380000403 to 0x380000406
LustreError: 11997:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800650f6440 x1833413952442240/t0(0) o104->lustre-MDT0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 11997:0:(client.c:1373:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
Lustre: ll_ost_out00_00: service thread pid 13221 completed after 120.340s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0001-mdc-ffff8802e5748008: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 6975:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e5748008: inode [0x240000404:0x55da:0x0] mdc close failed: rc = -5
LustreError: 6975:0:(file.c:247:ll_close_inode_openhandle()) Skipped 44 previous similar messages
LustreError: 2643:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x24000040a:0x890:0x0] error -108.
LustreError: 2643:0:(vvp_io.c:1909:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 10993:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 10993:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 3 previous similar messages
Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x340000402 to 0x340000405
LustreError: 17713:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 17713:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 2 previous similar messages
LustreError: 10965:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 10965:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 6 previous similar messages
LustreError: 24764:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802315d6940 x1833413991332352/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff8800a72fb7e8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3962:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x716:0x0]/ may get corrupted (rc -108)
Lustre: 3962:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040b:0x1a1:0x0]/ may get corrupted (rc -108)
Lustre: 3966:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x77b:0x0]/ may get corrupted (rc -108)
Lustre: 3966:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x765:0x0]/ may get corrupted (rc -108)
Lustre: 3951:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x782:0x0]// may get corrupted (rc -108)
Lustre: 3951:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xb00:0x0]/ may get corrupted (rc -108)
Lustre: 3958:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x6e9:0x0]/ may get corrupted (rc -108)
Lustre: 3958:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040b:0xf2:0x0]/ may get corrupted (rc -108)
Lustre: 3964:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000407:0x1c45:0x0]/ may get corrupted (rc -108)
Lustre: 3964:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040b:0x198:0x0]/ may get corrupted (rc -108)
Lustre: 3960:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x70f:0x0]// may get corrupted (rc -108)
Lustre: 3960:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040b:0x10e:0x0]/ may get corrupted (rc -108)
Lustre: 3963:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000408:0x735:0x0]// may get corrupted (rc -108)
LustreError: 19868:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748482509 with bad export cookie 17598892061532268963
LustreError: 14747:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff88028cdc1bf8 ns: filter-lustre-OST0000_UUID lock: ffff880272c9d2c0/0xf43bd316494e2eb8 lrc: 3/0,0 mode: PW/PW res: [0x2c0000404:0xceb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000080020000 nid: 0@lo remote: 0xf43bd316494e2eb1 expref: 3758 pid: 14747 timeout: 0 lvb_type: 0
LustreError: lustre-OST0000-osc-ffff8802e5748008: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3960:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000407:0x1b91:0x0]/ may get corrupted (rc -108)
Lustre: 3958:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x20000040a:0x60f:0x0]// may get corrupted (rc -108)
Lustre: 3951:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040b:0x1c2:0x0]// may get corrupted (rc -108)
Lustre: 3964:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xaef:0x0]/ may get corrupted (rc -108)
Lustre: 3961:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xa9f:0x0]/ may get corrupted (rc -108)
Lustre: 3961:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xa4e:0x0]/ may get corrupted (rc -108)
Lustre: 3960:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x20000040a:0x62b:0x0]/ may get corrupted (rc -108)
Lustre: 3964:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x24000040a:0xa93:0x0]/ may get corrupted (rc -108)
Lustre: 3954:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x280000407:0x1c5a:0x0]/ may get corrupted (rc -108)
Lustre: 3961:0:(llite_lib.c:4214:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x20000040a:0x631:0x0]/ may get corrupted (rc -108)
LustreError: 16736:0:(ldlm_resource.c:1097:ldlm_resource_complain()) lustre-OST0002-osc-ffff8800a72fb7e8: namespace resource [0x380000405:0x411:0x0].0x0 (ffff8800957ba340) refcount nonzero (0) after lock cleanup; forcing cleanup.
LustreError: 16736:0:(ldlm_resource.c:1097:ldlm_resource_complain()) Skipped 573 previous similar messages
Lustre: lustre-OST0000: haven't heard from client ebe878d2-f553-43ee-90ff-4d334c9ddb8a (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp ffff8802375dca88, cur 1748482541 deadline 1748482540 last 1748482510
Lustre: lustre-OST0000-osc-ffff8802e5748008: disconnect after 49s idle
LustreError: 18056:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 18056:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 5 previous similar messages
Lustre: 18248:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 6/24/0, destroy: 1/4/0
Lustre: 18248:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1673 previous similar messages
Lustre: 18248:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 3241/3241/0, xattr_set: 4861/45480/0
Lustre: 18248:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1673 previous similar messages
Lustre: 18248:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 28/157/0, punch: 0/0/0, quota 1/3/0
Lustre: 18248:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1673 previous similar messages
Lustre: 18248:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 7/118/0, delete: 2/5/1
Lustre: 18248:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1673 previous similar messages
Lustre: 18248:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 18248:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1673 previous similar messages
ptlrpc_watchdog_fire: 30 callbacks suppressed
Lustre: mdt_out00_004: service thread pid 23038 was inactive for 40.158 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 23038, comm: mdt_out00_004 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1844/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 11281:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-MDT0000: opcode 0: before 514 < left 3567, rollback = 0
Lustre: 11281:0:(osd_internal.h:1335:osd_trans_exec_op()) Skipped 1692 previous similar messages
LustreError: 23038:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880261270008 ns: mdt-lustre-MDT0001_UUID lock: ffff880236ae8f40/0xf43bd316497b429a lrc: 3/0,0 mode: PR/PR res: [0x24000040a:0xc0d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xf43bd31649673d66 expref: 40 pid: 23038 timeout: 0 lvb_type: 0
Lustre: mdt_out00_004: service thread pid 23038 completed after 99.811s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0001-mdc-ffff8802e5748008: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 30884:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000407:0x178e:0x0] doesn't exist!: rc = -14
LustreError: 30884:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 5 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2cdfd2067 PUD 28a9a6067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: llz4hc(OE) llz4(OE) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw libata i2c_core floppy
CPU: 7 PID: 8529 Comm: ll_sa_6481 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802c0c8b760 ti: ffff8802d9340000 task.ti: ffff8802d9340000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802d9343d78 EFLAGS: 00010296
RAX: ffff8801edfa0a90 RBX: 0000000000000008 RCX: 0000000100260025
RDX: 0000000000000026 RSI: ffff8801edfa0dc0 RDI: 0000000000000008
RBP: ffff8802d9343d88 R08: ffff88028b42dcc8 R09: 0000000000000000
R10: ffff88028b42e6b8 R11: ffff88028b42dcc8 R12: 0000000000000000
R13: ffff88025a88dc40 R14: ffff8801edfa0dc0 R15: ffff88028b42dcc8
FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000245318000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15d0bc1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa15cfe10>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

cp (28009) used greatest stack depth: 10064 bytes left
LustreError: 27893:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x200000403:0x7:0x0] mdc close failed: rc = -13
Lustre: 29343:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/11 is open, migrate only dentry
5[30758]: segfault at 0 ip 00007fe4cff2a59a sp 00007fff60633530 error 4 in ld-2.17.so[7fe4cff28000+22000]
Lustre: 15025:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/7 is open, migrate only dentry
Lustre: 15025:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 2 previous similar messages
Lustre: 30495:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x200000403:0x2:0x0]/12 is open, migrate only dentry
LustreError: 30630:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/5 failed: rc = -2
LustreError: 30630:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000403:0x3b:0x0]/15 failed: rc = -2
LustreError: 5879:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xa4:0x0]: rc = -5
LustreError: 5879:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 15025:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x9b:0x0]/10 failed: rc = -2
LustreError: 6312:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xa4:0x0]: rc = -5
LustreError: 6312:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 4810:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x20:0x0] with magic=0xbd60bd0
Lustre: 30630:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/1 is open, migrate only dentry
LustreError: 15025:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000404:0x28:0x0]/4 failed: rc = -2
Lustre: 7549:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802be786440 x1833361004283904/t4294970033(0) o101->7c54f921-d369-4f1e-8f05-a66b3b2a9418@0@lo:424/0 lens 376/816 e 0 to 0 dl 1748429424 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 30495:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000404:0xf4:0x0]/4 failed: rc = -2
LustreError: 30495:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 1 previous similar message
Lustre: 15027:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/2 is open, migrate only dentry
Lustre: 15027:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 1 previous similar message
Lustre: 28077:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880299aa4140 x1833361005493376/t4294970511(0) o101->413fca67-bccb-4767-b306-cead9605e937@0@lo:430/0 lens 376/840 e 0 to 0 dl 1748429430 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 10962:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xa4:0x0]: rc = -5
LustreError: 10962:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 10962:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 10962:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 4750:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x20:0x0] with magic=0xbd60bd0
Lustre: 4750:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 29901:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x174:0x0]/7 failed: rc = -2
LustreError: 29901:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 14293:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xa4:0x0]: rc = -5
LustreError: 14293:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 7430:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x280000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 7430:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 1 previous similar message
LustreError: 30495:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 17' to finish migration: rc = -1
LustreError: 15253:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x171:0x0]: rc = -5
LustreError: 15253:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 12389:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x240000403:0x10c:0x0] dir layout mismatch:
LustreError: 12389:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=3 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 12389:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0xd:0x0]
LustreError: 12389:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 15027:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x280000403:0x188:0x0]/11 failed: rc = -2
LustreError: 15027:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 3 previous similar messages
LustreError: 13578:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '0' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 0' to finish migration: rc = -1
Lustre: 4662:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802bbcfda40 x1833361010317184/t4294974386(0) o101->7c54f921-d369-4f1e-8f05-a66b3b2a9418@0@lo:463/0 lens 376/816 e 0 to 0 dl 1748429463 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 20486:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xa4:0x0]: rc = -5
LustreError: 20486:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 20486:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 20486:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 1 previous similar message
Lustre: dir [0x280000403:0x188:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
2[19194]: segfault at 8 ip 00007fc66f6eb7e8 sp 00007fff9b35faa0 error 4 in ld-2.17.so[7fc66f6e0000+22000]
Lustre: 29535:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/18 is open, migrate only dentry
Lustre: 29535:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 4 previous similar messages
LustreError: 29123:0:(mdt_reint.c:876:mdt_add_dirty_flag()) file attribute read error for [0x240000403:0x1f8:0x0]: -2.
LustreError: 29123:0:(osd_handler.c:211:osd_trans_start()) lustre-MDT0001: can't assign tx: rc = -2
LustreError: 22020:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x240000403:0x1f8:0x0] mdc close failed: rc = -2
LustreError: 19431:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x200000404:0x1e2:0x0] mdc close failed: rc = -2
15[24357]: segfault at 0 ip 0000000000403e5f sp 00007ffcb9d829a0 error 6 in 19[400000+6000]
LustreError: 28414:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x224:0x0]: rc = -5
LustreError: 28414:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 3 previous similar messages
LustreError: 28414:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 28414:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 3 previous similar messages
LustreError: 1042:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x240000403:0x23b:0x0] mdc close failed: rc = -2
LustreError: 4727:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '14' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 14' to finish migration: rc = -1
LustreError: 4727:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000403:0x1:0x0]/14 failed: rc = -1
LustreError: 4727:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 4 previous similar messages
Lustre: 4765:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x13e:0x0] with magic=0xbd60bd0
Lustre: 4765:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 30062:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/8 is open, migrate only dentry
Lustre: 30062:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 8 previous similar messages
LustreError: 28424:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0xde:0x0]: rc = -2
LustreError: 11198:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x240000403:0xde:0x0] mdc close failed: rc = -2
LustreError: 30074:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '13' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 13' to finish migration: rc = -1
LustreError: 15197:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x15a:0x0]: rc = -5
LustreError: 15197:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 19 previous similar messages
LustreError: 15197:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 15197:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 19 previous similar messages
LustreError: 15014:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0xde:0x0]: rc = -2
LustreError: 28734:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x489:0x0] migrate mdt count mismatch 3 != 2
LustreError: 10870:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x240000404:0x2b9:0x0] mdc close failed: rc = -2
LustreError: 10870:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 30470:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0xa2:0x0] migrate mdt count mismatch 2 != 3
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000403:0x409:0x0]/0xa): rc = 0
LustreError: 20222:0:(osd_index.c:204:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -2
LustreError: 215:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 5 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 16800:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cb819bf8: cannot apply new layout on [0x240000404:0x34d:0x0] : rc = -5
LustreError: 16800:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000404:0x34d:0x0] error -5.
LustreError: 4810:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x56d:0x0] migrate mdt count mismatch 2 != 3
LustreError: 4727:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 17' to finish migration: rc = -1
LustreError: 4727:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 1 previous similar message
1[29956]: segfault at 0 ip 0000000000403e5f sp 00007ffec27b3f00 error 6 in 1[400000+6000]
LustreError: 29351:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cb819bf8: cannot apply new layout on [0x240000404:0x34d:0x0] : rc = -5
LustreError: 29351:0:(lov_object.c:1350:lov_layout_change()) Skipped 2 previous similar messages
Lustre: dir [0x280000404:0x307:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 29535:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '11' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 11' to finish migration: rc = -1
Lustre: dir [0x240000404:0x53f:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 383:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 10 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
Lustre: 29535:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x569:0x0] with magic=0xbd60bd0
Lustre: 29535:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000404:0x54c:0x0]/0xa): rc = 0
LustreError: 7890:0:(osd_index.c:204:__osd_xattr_load_by_oid()) lustre-MDT0001: can't get bonus, rc = -2
LustreError: 26632:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x3ef:0x0]/15 failed: rc = -116
LustreError: 26632:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 9 previous similar messages
LustreError: 4770:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
Lustre: 29535:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x54d:0x0] with magic=0xbd60bd0
Lustre: 29535:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 1 previous similar message
19[16475]: segfault at 8 ip 00007fe8775467e8 sp 00007ffca8ed4450 error 4 in ld-2.17.so[7fe87753b000+22000]
LustreError: 27882:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0xa2:0x0] migrate mdt count mismatch 2 != 3
LustreError: 12995:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x240000404:0x61a:0x0] dir layout mismatch:
LustreError: 12995:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 12995:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x2a:0x0]
LustreError: 12995:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 6 previous similar messages
LustreError: 12995:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 30495:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '6' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 6' to finish migration: rc = -1
Lustre: 7430:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x62d:0x0]/14 is open, migrate only dentry
Lustre: 7430:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 14 previous similar messages
Lustre: 13044:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x469:0x0] with magic=0xbd60bd0
Lustre: 13044:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 7 previous similar messages
LustreError: 20863:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x6e7:0x0]: rc = -5
LustreError: 20863:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 32 previous similar messages
LustreError: 20863:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 20863:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 32 previous similar messages
Lustre: dir [0x200000404:0x643:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 863:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 863:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 14634:0:(statahead.c:2447:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: dir [0x240000403:0x570:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x833:0x0] with magic=0xbd60bd0
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 9 previous similar messages
LustreError: 4625:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x74e:0x0] migrate mdt count mismatch 2 != 1
LustreError: 31693:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x240000403:0x45a:0x0] : rc = -5
LustreError: 28521:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x200000404:0x688:0x0] mdc close failed: rc = -2
Lustre: dir [0x240000404:0x724:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 9 previous similar messages
LustreError: 29544:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '8' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 8' to finish migration: rc = -1
LustreError: 29544:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
LustreError: 408:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 408:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 15 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x663:0x0] with magic=0xbd60bd0
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 47 previous similar messages
LustreError: 38:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 13 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 17888:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x240000404:0x8ef:0x0] mdc close failed: rc = -2
LustreError: 17888:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 25784:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x200000403:0xae8:0x0] : rc = -5
LustreError: 25784:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000403:0xae8:0x0] error -5.
LustreError: 25784:0:(vvp_io.c:1909:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 7494:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x878:0x0] migrate mdt count mismatch 3 != 1
LustreError: 2169:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cb819bf8: cannot apply new layout on [0x240000404:0xb27:0x0] : rc = -5
LustreError: 2169:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000404:0xb27:0x0] error -5.
LustreError: 12873:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x200000403:0xae8:0x0] : rc = -5
LustreError: 256:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 256:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 28731:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0xd13:0x0] migrate mdt count mismatch 2 != 1
Lustre: dir [0x240000404:0x956:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 6 previous similar messages
8[22291]: segfault at 8 ip 00007f596de777e8 sp 00007ffd849d8690 error 4 in ld-2.17.so[7f596de6c000+22000]
LustreError: 408:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x83d:0x0]: rc = -5
LustreError: 408:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 86 previous similar messages
LustreError: 408:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 408:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 86 previous similar messages
LustreError: 408:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
Lustre: mdt00_031: service thread pid 30275 was inactive for 40.106 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 30275, comm: mdt00_031 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_setattr+0x1324/0x15f0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 30218, comm: mdt00_029 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_object_open_lock+0x6b9/0xc10 [mdt]
[<0>] mdt_open_by_fid_lock+0x4ec/0xc10 [mdt]
[<0>] mdt_reint_open+0x9d8/0x2d70 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_intent_open+0x93/0x480 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_045: service thread pid 30519 was inactive for 40.098 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 30519, comm: mdt00_045 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xabc/0xfd0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xb00 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_pdo_lock+0x729/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_reint_unlink+0x187/0x1610 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x23d/0xd10 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io00_021: service thread pid 13044 was inactive for 72.228 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io00_013: service thread pid 13673 was inactive for 72.105 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: mdt_io00_019: service thread pid 4727 was inactive for 72.114 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
Lustre: mdt_io00_003: service thread pid 29343 was inactive for 72.102 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff880245568040/0x2d318c70316b9438 lrc: 3/0,0 mode: PW/PW res: [0x280000404:0xc62:0x0].0x0 bits 0x4/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c70316b942a expref: 447 pid: 7708 timeout: 517 lvb_type: 0
Lustre: mdt00_031: service thread pid 30275 completed after 100.401s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 13044:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/4 is open, migrate only dentry
Lustre: 13044:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 35 previous similar messages
Lustre: mdt00_029: service thread pid 30218 completed after 100.164s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
Lustre: mdt_io00_021: service thread pid 13044 completed after 100.407s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 19864:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x280000403:0x7b9:0x0] mdc close failed: rc = -108
LustreError: 23163:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
Lustre: mdt_io00_017: service thread pid 30062 completed after 99.860s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_013: service thread pid 13673 completed after 99.895s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_006: service thread pid 30074 completed after 99.803s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_012: service thread pid 13578 completed after 99.765s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_010: service thread pid 30630 completed after 99.971s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_016: service thread pid 29978 completed after 99.795s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_020: service thread pid 12941 completed after 99.255s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_009: service thread pid 30594 completed after 99.233s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_000: service thread pid 15025 completed after 98.872s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_019: service thread pid 4727 completed after 98.780s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_011: service thread pid 7430 completed after 98.314s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_007: service thread pid 30495 completed after 97.979s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_014: service thread pid 26632 completed after 97.962s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 15027:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '2' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 2' to finish migration: rc = -1
LustreError: 15027:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 9 previous similar messages
LustreError: 15027:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000403:0x1:0x0]/2 failed: rc = -1
LustreError: 15027:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 32 previous similar messages
Lustre: mdt_io00_002: service thread pid 15027 completed after 97.939s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_045: service thread pid 30519 completed after 98.313s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x7fa:0x0] with magic=0xbd60bd0
Lustre: 3309:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 33 previous similar messages
Lustre: mdt_io00_004: service thread pid 29535 completed after 98.616s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_008: service thread pid 30549 completed after 98.560s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_003: service thread pid 29343 completed after 97.766s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_015: service thread pid 29544 completed after 97.450s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_022: service thread pid 13111 completed after 96.360s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 7460:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
Lustre: dir [0x240000403:0xcb2:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
19[17331]: segfault at 8 ip 00007fc2fa17c7e8 sp 00007fffb750bac0 error 4 in ld-2.17.so[7fc2fa171000+22000]
LustreError: 4754:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0xf59:0x0] ACL: rc = -2
8[32128]: segfault at 8 ip 00007fdf54be27e8 sp 00007fffb2e0a9d0 error 4 in ld-2.17.so[7fdf54bd7000+22000]
LustreError: 30074:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 28174:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0xd65:0x0] ACL: rc = -2
9[21268]: segfault at 0 ip (null) sp 00007ffdd72932c8 error 14 in 9[400000+6000]
5[6379]: segfault at 8 ip 00007f1f910c07e8 sp 00007ffc970f8bf0 error 4 in ld-2.17.so[7f1f910b5000+22000]
19[9347]: segfault at 8 ip 00007fbd576707e8 sp 00007ffc50088e90 error 4 in ld-2.17.so[7fbd57665000+22000]
Lustre: dir [0x280000404:0x1152:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 7712:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x1643:0x0] migrate mdt count mismatch 3 != 2
LustreError: 11157:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x200000403:0x14ee:0x0] mdc close failed: rc = -2
LustreError: 11157:0:(file.c:247:ll_close_inode_openhandle()) Skipped 20 previous similar messages
LustreError: 127:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x240000404:0x1478:0x0] inode@0000000000000000: rc = -5
LustreError: 127:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 18678:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x280000405:0x43d:0x0] : rc = -5
LustreError: 18678:0:(lov_object.c:1350:lov_layout_change()) Skipped 1 previous similar message
LustreError: 18678:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000405:0x43d:0x0] error -5.
13[27033]: segfault at 8 ip 00007ffa8824e7e8 sp 00007ffcd1c27110 error 4 in ld-2.17.so[7ffa88243000+22000]
Lustre: 7453:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x1220:0x0] with magic=0xbd60bd0
Lustre: 7453:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 35 previous similar messages
LustreError: 4727:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 5' to finish migration: rc = -1
LustreError: 4727:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 13 previous similar messages
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88030edf2200/0x2d318c7031792aba lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x133f:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c7031792a97 expref: 865 pid: 30214 timeout: 666 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802caabdd28: operation mds_hsm_state_set to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802caabdd28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802caabdd28: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 32561:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000404:0x133f:0x0] error -5.
LustreError: 7069:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 6988:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0x1862:0x0] error: rc = -108
LustreError: 7202:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x200000403:0x154b:0x0] get parent: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8802caabdd28: Connection restored to 192.168.123.67@tcp (at 0@lo)
LustreError: 30447:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ca931bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff880284be07c0/0x2d318c7031905844 lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x1643:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2d318c7031905836 expref: 9 pid: 30447 timeout: 0 lvb_type: 0
2[11004]: segfault at 8 ip 00007f8206ca17e8 sp 00007ffd1ac48cc0 error 4 in ld-2.17.so[7f8206c96000+22000]
INFO: task mrename:858 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff88029e1249d0 11696 858 27459 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:2775 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff880099e9ca80 11696 2775 27528 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:4152 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802b104aae8 10736 4152 27298 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:4227 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802c158aae8 11696 4227 27166 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:5278 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802bcbf52c0 11696 5278 27423 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:8496 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff88009016bd58 11696 8496 27446 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:8580 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802be654a80 10928 8580 27337 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:9145 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802c25c9280 11696 9145 27393 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:10243 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff88009ab53d58 11808 10243 27280 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task mrename:11854 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mrename D ffff8802c7583a00 11696 11854 27579 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff81252061>] lock_rename+0x31/0xe0
[<ffffffff812586af>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed462>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff812597de>] SyS_renameat2+0xe/0x10
[<ffffffff8125981e>] SyS_rename+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802cda05e00/0x2d318c70318eca32 lrc: 3/0,0 mode: PR/PR res: [0x280000405:0x803:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c70318eca16 expref: 708 pid: 30339 timeout: 761 lvb_type: 0
LustreError: lustre-MDT0002-mdc-ffff8802caabdd28: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802caabdd28: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 20425:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748430022 with bad export cookie 3256538419055133309
LustreError: lustre-MDT0002-mdc-ffff8802caabdd28: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 14077:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0002-mdc-ffff8802caabdd28: [0x280000401:0x2a:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0002-mdc-ffff8802caabdd28: Connection restored to 192.168.123.67@tcp (at 0@lo)
LustreError: 532:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0xd92:0x0]: rc = -5
LustreError: 532:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 78 previous similar messages
LustreError: 532:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 532:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 78 previous similar messages
LustreError: 30495:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 4625:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x1690:0x0] migrate mdt count mismatch 1 != 3
8[24613]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffca0edd110 error 14
LustreError: 30495:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/4 failed: rc = -1
LustreError: 30495:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 49 previous similar messages
Lustre: 29901:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/6 is open, migrate only dentry
Lustre: 29901:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 55 previous similar messages
LustreError: 45:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 14 [0x200000403:0x15a2:0x0] inode@0000000000000000: rc = -5
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000405:0x94b:0x0]/0xa): rc = 0
LustreError: 12689:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x280000405:0x8a5:0x0] : rc = -5
LustreError: 12689:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000405:0x8a5:0x0] error -5.
Lustre: dir [0x280000406:0x136:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 26 previous similar messages
LustreError: 7682:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x240000404:0x1659:0x0] error -5.
LustreError: 7682:0:(vvp_io.c:1909:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 4759:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000400:0x5b:0x0] ACL: rc = -2
3[22091]: segfault at 8 ip 00007fcafe3507e8 sp 00007ffeb99edb30 error 4 in ld-2.17.so[7fcafe345000+22000]
LustreError: 28333:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000405:0x11e:0x0]: rc = -2
LustreError: 15211:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x200000403:0x15a2:0x0] : rc = -5
LustreError: 15211:0:(lov_object.c:1350:lov_layout_change()) Skipped 7 previous similar messages
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff880099db96c0/0x2d318c7031acc725 lrc: 3/0,0 mode: PR/PR res: [0x280000406:0x374:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c7031acc6fb expref: 529 pid: 4707 timeout: 974 lvb_type: 0
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: operation mds_getattr_lock to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 4 previous similar messages
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 24927:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cb819bf8: inode [0x280000405:0xdda:0x0] mdc close failed: rc = -108
LustreError: 28208:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x19fa:0x0] error: rc = -5
LustreError: 28208:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 158 previous similar messages
LustreError: 24927:0:(file.c:247:ll_close_inode_openhandle()) Skipped 74 previous similar messages
LustreError: 25969:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0002-mdc-ffff8802cb819bf8: [0x280000402:0x5c:0x0] lock enqueue fails: rc = -108
LustreError: 25969:0:(mdc_request.c:1469:mdc_read_page()) Skipped 18 previous similar messages
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
Lustre: 6473:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1d19:0x0] with magic=0xbd60bd0
Lustre: 6473:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 33 previous similar messages
LustreError: 13458:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '5' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 5' to finish migration: rc = -1
LustreError: 13458:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 21 previous similar messages
17[13281]: segfault at 8 ip 00007fc68a35a7e8 sp 00007ffd75ea85a0 error 4 in ld-2.17.so[7fc68a34f000+22000]
3[19986]: segfault at 0 ip (null) sp 00007ffc5ce24dc8 error 14 in 3[400000+6000]
LustreError: 7460:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x1a59:0x0] migrate mdt count mismatch 1 != 3
LustreError: 7460:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 3 previous similar messages
8[26626]: segfault at 8 ip 00007f5082a6b7e8 sp 00007ffcc62d9eb0 error 4 in ld-2.17.so[7f5082a60000+22000]
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802e41ba200/0x2d318c7031b9541e lrc: 3/0,0 mode: PR/PR res: [0x280000406:0x953:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c7031b95402 expref: 139 pid: 28032 timeout: 1111 lvb_type: 0
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 5119:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000406:0x953:0x0] error: rc = -5
LustreError: 7255:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 7255:0:(llite_lib.c:2023:ll_md_setattr()) Skipped 3 previous similar messages
LustreError: 5119:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 8 previous similar messages
Lustre: lustre-MDT0002-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
LustreError: 12941:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
15[8635]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffc88e6fac0 error 14
Lustre: dir [0x200000403:0x1ee8:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 11 previous similar messages
LustreError: 15015:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x197e:0x0]: rc = -2
LustreError: 15015:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 7502:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
LustreError: 28201:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
LustreError: 7453:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
LustreError: 31:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 9 [0x280000408:0x1f:0x0] inode@0000000000000000: rc = -5
LustreError: 31:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 7 previous similar messages
15[6838]: segfault at 8 ip 00007ff4392c27e8 sp 00007ffcf4c5f0f0 error 4 in ld-2.17.so[7ff4392b7000+22000]
LustreError: 31172:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000406:0x7ff:0x0]: rc = -2
LustreError: 31172:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 26425:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caabdd28: cannot apply new layout on [0x200000403:0x236b:0x0] : rc = -5
LustreError: 26425:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000403:0x236b:0x0] error -5.
LustreError: 31157:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000408:0xb1:0x0]: rc = -2
LustreError: 31157:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
15[1051]: segfault at 8 ip 00007f23c283f7e8 sp 00007ffd0af09730 error 4 in ld-2.17.so[7f23c2834000+22000]
LustreError: 16153:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x280000408:0x26b:0x0] dir layout mismatch:
LustreError: 16153:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 16153:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x280000400:0x6d:0x0]
LustreError: 16153:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 6 previous similar messages
LustreError: 16153:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=6 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:3 pool=
LustreError: 4822:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
LustreError: 4822:0:(mdt_open.c:1703:mdt_reint_open()) Skipped 1 previous similar message
11[5862]: segfault at 8 ip 00007f962c09c7e8 sp 00007fffe83c3090 error 4 in ld-2.17.so[7f962c091000+22000]
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88028a11a980/0x2d318c7031e41316 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x2484:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c7031e412ec expref: 523 pid: 7549 timeout: 1320 lvb_type: 0
Lustre: 30062:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x246c:0x0]/17 is open, migrate only dentry
Lustre: 30062:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 112 previous similar messages
LustreError: 471:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748430582 with bad export cookie 3256538419055133064
Lustre: lustre-MDT0001-mdc-ffff8802cb819bf8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff8802cb819bf8: operation mds_getattr_lock to node 0@lo failed: rc = -107
LustreError: lustre-MDT0001-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
Lustre: lustre-MDT0001-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
LustreError: 13673:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000403:0x2311:0x0]/15 failed: rc = -2
LustreError: 13673:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 74 previous similar messages
14[16110]: segfault at 0 ip 0000000000403e5f sp 00007fffa6b30c70 error 6 in 14[400000+6000]
8[18187]: segfault at 8 ip 00007fedcd4777e8 sp 00007ffca0eefb40 error 4 in ld-2.17.so[7fedcd46c000+22000]
LustreError: 15217:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000405:0xaf:0x0]: rc = -5
LustreError: 15217:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 508 previous similar messages
LustreError: 15217:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 15217:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 508 previous similar messages
LustreError: 13416:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x200000403:0x26b3:0x0] dir layout mismatch:
LustreError: 13416:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13416:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0x9d:0x0]
LustreError: 13416:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 8 previous similar messages
LustreError: 13416:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 2517:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000408:0x3d2:0x0] error -5.
LustreError: 13419:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13419:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 13425:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13425:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 13422:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13422:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 13209:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13209:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 13206:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13206:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
LustreError: 13204:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 13204:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:2000003 pool=
16[17306]: segfault at 0 ip (null) sp 00007fffc4c50bf8 error 14 in 16[400000+6000]
LustreError: 32019:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x24b2:0x0]: rc = -2
LustreError: 32019:0:(mdd_object.c:3901:mdd_close()) Skipped 3 previous similar messages
LustreError: 28357:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000406:0x118c:0x0] migrate mdt count mismatch 1 != 3
LustreError: 28357:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 7 previous similar messages
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880277ebda40/0x2d318c7031ec3144 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x2732:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c7031ec3128 expref: 684 pid: 3309 timeout: 1454 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802cb819bf8: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802cb819bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 16411:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000408:0xae3:0x0] error: rc = -5
LustreError: 16411:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 1 previous similar message
LustreError: 17180:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 8955:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8802cb819bf8: [0x200000402:0x6c:0x0] lock enqueue fails: rc = -5
LustreError: 8955:0:(mdc_request.c:1469:mdc_read_page()) Skipped 3 previous similar messages
LustreError: 18389:0:(ldlm_resource.c:1097:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8802cb819bf8: namespace resource [0x200000403:0x2b1e:0x0].0x0 (ffff880296b434c0) refcount nonzero (0) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
LustreError: 6013:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802cb81d3d8 ns: mdt-lustre-MDT0000_UUID lock: ffff88030ef570c0/0x2d318c7031fd6627 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2d318c7031fd660b expref: 20 pid: 6013 timeout: 0 lvb_type: 0
Lustre: 12550:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0002: [0x280000403:0x1:0x0]/12 is open, migrate only dentry
LustreError: 7502:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '13' present, but FID [0x200000404:0xc9c:0x0] is invalid
LustreError: 12941:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '5' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 5' to finish migration: rc = -1
LustreError: 12941:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 22 previous similar messages
Lustre: 30495:0:(lod_lov.c:1403:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2464:0x0] with magic=0xbd60bd0
Lustre: 30495:0:(lod_lov.c:1403:lod_parse_striping()) Skipped 157 previous similar messages
LustreError: 1091:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caabdd28: inode [0x200000406:0xe0:0x0] mdc close failed: rc = -2
LustreError: 1091:0:(file.c:247:ll_close_inode_openhandle()) Skipped 125 previous similar messages
15[7919]: segfault at 8 ip 00007febb860c7e8 sp 00007ffd20a79720 error 4 in ld-2.17.so[7febb8601000+22000]
3[5170]: segfault at 0 ip (null) sp 00007ffdeb222fa8 error 14 in 3[400000+6000]
3[5130]: segfault at 0 ip (null) sp 00007ffe3556ac78 error 14 in 3[400000+6000]
13[2218]: segfault at 8 ip 00007f3ba246d7e8 sp 00007ffc48ed6640 error 4 in ld-2.17.so[7f3ba2462000+22000]
LustreError: 31793:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X0.
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8800adfcc780/0x2d318c7032088608 lrc: 3/0,0 mode: PR/PR res: [0x200000406:0x299:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c703208858a expref: 232 pid: 30218 timeout: 1618 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802cb819bf8: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802cb819bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 17328:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
8[23679]: segfault at 8 ip 00007f42819b77e8 sp 00007ffc9473fd70 error 4 in ld-2.17.so[7f42819ac000+22000]
Lustre: dir [0x280000406:0x1577:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 74 previous similar messages
LustreError: 26719:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cb819bf8: cannot apply new layout on [0x280000406:0x14ab:0x0] : rc = -5
LustreError: 26719:0:(lov_object.c:1350:lov_layout_change()) Skipped 6 previous similar messages
LustreError: 26719:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x280000406:0x14ab:0x0] error -5.
LustreError: 26719:0:(vvp_io.c:1909:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 406:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 20 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 406:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 3 previous similar messages
16[12717]: segfault at 0 ip 0000000000403e5f sp 00007fffccad8470 error 6 in 16[400000+6000]
8[11079]: segfault at 8 ip 00007f558c48c7e8 sp 00007ffe764c3d90 error 4 in ld-2.17.so[7f558c481000+22000]
14[21945]: segfault at 8 ip 00007f2872f167e8 sp 00007ffcc75b5240 error 4 in ld-2.17.so[7f2872f0b000+22000]
1[7151]: segfault at 8 ip 00007f03ae2967e8 sp 00007ffdc47bcf30 error 4 in ld-2.17.so[7f03ae28b000+22000]
LustreError: 10198:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x200000407:0xc38:0x0] dir layout mismatch:
LustreError: 10198:0:(llite_lib.c:1872:ll_update_lsm_md()) Skipped 6 previous similar messages
LustreError: 10198:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=3 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 10198:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0xeb:0x0]
LustreError: 10198:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 41 previous similar messages
LustreError: 10198:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 18378:0:(vvp_io.c:1909:vvp_io_init()) lustre: refresh file layout [0x200000407:0xc26:0x0] error -5.
LustreError: 18378:0:(vvp_io.c:1909:vvp_io_init()) Skipped 5 previous similar messages
LustreError: 14836:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff880280b73880/0x2d318c70322b3a27 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x3c42:0x0].0x0 bits 0x12/0x0 rrc: 17 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2d318c70322b39fd expref: 698 pid: 7549 timeout: 1921 lvb_type: 0
LustreError: lustre-MDT0001-mdc-ffff8802cb819bf8: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff8802cb819bf8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 6013:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880243874a88 ns: mdt-lustre-MDT0001_UUID lock: ffff8802d320fc00/0x2d318c70323adfe0 lrc: 3/0,0 mode: PR/PR res: [0x240000405:0x1c87:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2d318c70323adfc4 expref: 490 pid: 6013 timeout: 0 lvb_type: 0
LustreError: 6013:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 3 previous similar messages
LustreError: lustre-MDT0001-mdc-ffff8802cb819bf8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 16572:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 15122:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x3c42:0x0] error: rc = -108
LustreError: 15122:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 173 previous similar messages
LustreError: 19977:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff8802cb819bf8: [0x240000400:0xbc:0x0] lock enqueue fails: rc = -108
LustreError: 19977:0:(mdc_request.c:1469:mdc_read_page()) Skipped 8 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff8802cb819bf8: Connection restored to 192.168.123.67@tcp (at 0@lo)
4[32316]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffdddab08f0 error 14
LustreError: 28193:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff880243874a88 ns: mdt-lustre-MDT0001_UUID lock: ffff88008fd034c0/0x2d318c70323ade58 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0xda:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2d318c70323ade3c expref: 3 pid: 28193 timeout: 0 lvb_type: 0
Lustre: 12941:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/3 is open, migrate only dentry
Lustre: 12941:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 145 previous similar messages
LustreError: 30549:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000405:0x9cb:0x0]/17 failed: rc = -2
LustreError: 30549:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 112 previous similar messages
LustreError: 7494:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000405:0x9cb:0x0] migrate mdt count mismatch 1 != 2
LustreError: 7494:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 7 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2a16d4067 PUD 2e633f067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix i2c_core serio_raw floppy virtio_blk libata [last unloaded: libcfs]
CPU: 10 PID: 27545 Comm: ll_sa_27303 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800a3bfc9d0 ti: ffff880324198000 task.ti: ffff880324198000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88032419bd78 EFLAGS: 00010296
RAX: ffff8802bcb03a10 RBX: 0000000000000008 RCX: 0000000100260022
RDX: 0000000000000026 RSI: ffff8802bcb03d40 RDI: 0000000000000008
RBP: ffff88032419bd88 R08: ffff8802f2d7c8e8 R09: 0000000000000000
R10: ffff8802f2d7d978 R11: ffff8802f2d7c8e8 R12: 0000000000000000
R13: ffff8802f2a6dc40 R14: ffff8802bcb03d40 R15: ffff8802f2d7c8e8
FS: 0000000000000000(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000326ce2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa176cbb1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa176be00>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 32095:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a9dcf340 x1832974231584640/t4294967609(0) o101->8c6aa17c-7289-4045-8354-85e0c248f99d@0@lo:713/0 lens 376/816 e 0 to 0 dl 1748060518 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
7[3836]: segfault at 8 ip 00007fd2002fe7e8 sp 00007ffe86db6e10 error 4 in ld-2.17.so[7fd2002f3000+22000]
Lustre: 32364:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32364:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 32364:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 32364:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 32364:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 32364:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
10[7386]: segfault at 8 ip 00007fbb553067e8 sp 00007ffeca8eab60 error 4 in ld-2.17.so[7fbb552fb000+22000]
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 9 previous similar messages
LustreError: 8439:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0f2efc8: inode [0x200000401:0x240:0x0] mdc close failed: rc = -13
12[11084]: segfault at 0 ip (null) sp 00007ffe11861598 error 14 in 12[400000+6000]
Lustre: 1887:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 588, rollback = 7
Lustre: 1887:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1887:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1887:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1887:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1887:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1887:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/588/0, punch: 0/0/0, quota 1/3/0
Lustre: 1887:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1887:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1887:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1887:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1887:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
1[13319]: segfault at 8 ip 00007f7b9e2737e8 sp 00007ffe1e9fcc90 error 4 in ld-2.17.so[7f7b9e268000+22000]
Lustre: 3550:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 3550:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3550:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3550:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3550:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3550:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3550:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 3550:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3550:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3550:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3550:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3550:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
2[14415]: segfault at 406000 ip 0000000000406000 sp 00007ffff03a9598 error 14 in 2[606000+1000]
2[14635]: segfault at 8 ip 00007f73c13667e8 sp 00007ffe2df18b80 error 4 in ld-2.17.so[7f73c135b000+22000]
13[14627]: segfault at 0 ip 0000000000403e5f sp 00007ffde761b1b0 error 6 in 13[400000+6000]
LustreError: 16651:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800bb91dd28: inode [0x200000402:0x49d:0x0] mdc close failed: rc = -13
17[17259]: segfault at 406000 ip 0000000000406000 sp 00007ffdf6f860f8 error 14 in 17[606000+1000]
9[18141]: segfault at 8 ip 00007fad99f957e8 sp 00007ffe0f10eb60 error 4 in ld-2.17.so[7fad99f8a000+22000]
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 32366:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 32366:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 32366:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 32366:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 32366:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 32366:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 9 previous similar messages
2[23488]: segfault at 1c ip 00000000004017b0 sp 00007fffb47d9fb0 error 6 in 2[400000+6000]
Lustre: 3550:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 3550:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 25 previous similar messages
Lustre: 3550:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3550:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3550:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3550:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3550:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 3550:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3550:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3550:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3550:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3550:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 25 previous similar messages
5[24948]: segfault at 0 ip (null) sp 00007fff68a109a8 error 14 in 5[400000+6000]
6[26264]: segfault at 8 ip 00007f9d089637e8 sp 00007ffd3e1b5050 error 4 in ld-2.17.so[7f9d08958000+22000]
LustreError: 28889:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0f2efc8: inode [0x200000402:0x8c8:0x0] mdc close failed: rc = -13
5[32568]: segfault at 0 ip 00000000004043f0 sp 00007ffcf20f0278 error 6 in 16[400000+6000]
10[2127]: segfault at 8 ip 00007feab74887e8 sp 00007ffdc42d12f0 error 4 in ld-2.17.so[7feab747d000+22000]
Lustre: 18334:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18334:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 31 previous similar messages
Lustre: 18334:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18334:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 18334:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18334:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 18334:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 18334:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 18334:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18334:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 18334:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18334:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 31 previous similar messages
LustreError: 5544:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800bb91dd28: inode [0x200000402:0xbb8:0x0] mdc close failed: rc = -13
17[6018]: segfault at 8 ip 00007f573c77a7e8 sp 00007ffd130e6970 error 4 in ld-2.17.so[7f573c76f000+22000]
11[10298]: segfault at 8 ip 00007f160570f7e8 sp 00007ffcc48ba690 error 4 in ld-2.17.so[7f1605704000+22000]
1[13443]: segfault at 8 ip 00007f21cf2767e8 sp 00007ffd436b23f0 error 4 in ld-2.17.so[7f21cf26b000+22000]
15[14423]: segfault at 8 ip 00007f0e1c2bc7e8 sp 00007ffc9f7934c0 error 4 in ld-2.17.so[7f0e1c2b1000+22000]
15[16398]: segfault at 8 ip 00007f6a0ac707e8 sp 00007ffde5401680 error 4 in ld-2.17.so[7f6a0ac65000+22000]
traps: 18[20200] general protection ip:7f74d42a3bbb sp:7ffe4d3bda00 error:0 in ld-2.17.so[7f74d429c000+22000]
13[27786]: segfault at 8 ip 00007ff4536e17e8 sp 00007ffd74d17320 error 4 in ld-2.17.so[7ff4536d6000+22000]
Lustre: 3972:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1358:0x0] with magic=0xbd60bd0
Lustre: 1887:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 1887:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 71 previous similar messages
Lustre: 1887:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1887:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 1887:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1887:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 1887:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 1887:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 1887:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1887:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 1887:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1887:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 71 previous similar messages
LustreError: 2357:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800bb91dd28: inode [0x200000402:0x1599:0x0] mdc close failed: rc = -13
1[6251]: segfault at 8 ip 00007f2124f9e7e8 sp 00007ffdc08dd720 error 4 in ld-2.17.so[7f2124f93000+22000]
4[8564]: segfault at 0 ip 00000000004016cd sp 00007ffce7aff870 error 6 in 4[400000+6000]
19[9524]: segfault at 8 ip 00007f85b7bf07e8 sp 00007ffecc212c60 error 4 in ld-2.17.so[7f85b7be5000+22000]
13[10259]: segfault at 8 ip 00007f2da41d67e8 sp 00007ffe799240d0 error 4 in ld-2.17.so[7f2da41cb000+22000]
13[10522]: segfault at 0 ip (null) sp 00007fffa3d46de8 error 14 in 13[400000+6000]
0[16728]: segfault at 8 ip 00007f78290467e8 sp 00007ffcbe2fd090 error 4 in ld-2.17.so[7f782903b000+22000]
LustreError: 16728:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0f2efc8: inode [0x200000402:0x1a72:0x0] mdc close failed: rc = -13
Lustre: 14701:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1c7a:0x0] with magic=0xbd60bd0
Lustre: 14701:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
19[24058]: segfault at 8 ip 00007efdad4fa7e8 sp 00007ffca5bcd840 error 4 in ld-2.17.so[7efdad4ef000+22000]
10[27416]: segfault at 0 ip 0000000000403e5f sp 00007ffd2c343e30 error 6 in 10[400000+6000]
4[28046]: segfault at 8 ip 00007fb1cd90b7e8 sp 00007ffdd2735160 error 4 in ld-2.17.so[7fb1cd900000+22000]
3[30127]: segfault at 8 ip 00007f482b53e7e8 sp 00007ffc56e96b40 error 4 in ld-2.17.so[7f482b533000+22000]
LustreError: 31432:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X0.
Lustre: 1871:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x2086:0x0] with magic=0xbd60bd0
Lustre: 1871:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
0[5865]: segfault at 8 ip 00007fbc14f7b7e8 sp 00007ffc9d4e7970 error 4 in ld-2.17.so[7fbc14f70000+22000]
17[6758]: segfault at 8 ip 00007fc01ff797e8 sp 00007ffc723c0860 error 4 in ld-2.17.so[7fc01ff6e000+22000]
Lustre: 2264:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x227d:0x0] with magic=0xbd60bd0
Lustre: 2264:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: mdt00_011: service thread pid 4127 was inactive for 40.095 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 32474, comm: mdt00_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 1871, comm: mdt00_006 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_object_find_lock+0x54/0x170 [mdt]
[<0>] mdt_reint_setxattr+0x133/0x10e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 2 previous similar messages
Pid: 4127, comm: mdt00_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_intent_getxattr+0x80/0x330 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_012: service thread pid 5630 was inactive for 40.074 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 32087:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802f4238400/0x27e01505ef8b3cdc lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x2283:0x0].0x0 bits 0x1b/0x0 rrc: 14 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x27e01505ef8b3cb2 expref: 1119 pid: 32095 timeout: 3509 lvb_type: 0
Lustre: mdt00_006: service thread pid 1871 completed after 100.034s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 4127 completed after 100.018s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 32474 completed after 99.945s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800bb91dd28: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800bb91dd28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
Lustre: mdt00_010: service thread pid 3972 completed after 99.924s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800bb91dd28: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_012: service thread pid 5630 completed after 99.895s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 2264 completed after 99.559s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 14170 completed after 99.580s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 14168 completed after 99.476s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_013: service thread pid 10875 completed after 99.834s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 32096 completed after 99.552s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 8231:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 8208:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800bb91dd28: inode [0x200000401:0x21ee:0x0] mdc close failed: rc = -108
LustreError: 7855:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8800bb91dd28: [0x200000401:0x1:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8800bb91dd28: Connection restored to (at 0@lo)
4[8419]: segfault at 0 ip 0000000000403e5f sp 00007fff646d03f0 error 6 in 4[400000+6000]
Lustre: 6215:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 6215:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 201 previous similar messages
Lustre: 6215:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6215:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 201 previous similar messages
Lustre: 6215:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6215:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 201 previous similar messages
Lustre: 6215:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 6215:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 201 previous similar messages
Lustre: 6215:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6215:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 201 previous similar messages
Lustre: 6215:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6215:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 201 previous similar messages
Lustre: 3043:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x160:0x0] with magic=0xbd60bd0
Lustre: 3043:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
8[14700]: segfault at 8 ip 00007f98f33487e8 sp 00007ffd3e171be0 error 4 in ld-2.17.so[7f98f333d000+22000]
Lustre: 32474:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x2525:0x0] with magic=0xbd60bd0
Lustre: 32474:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 17052:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800bb91dd28: inode [0x200000402:0x2551:0x0] mdc close failed: rc = -13
LustreError: 17052:0:(file.c:247:ll_close_inode_openhandle()) Skipped 3 previous similar messages
traps: 4[23610] trap invalid opcode ip:4059da sp:7ffea1b93778 error:0 in 4[400000+6000]
10[31241]: segfault at 1c ip 00000000004017b0 sp 00007fff42e64b90 error 6 in 9 (deleted)[400000+6000]
0[32651]: segfault at 0 ip (null) sp 00007ffc358fe6c8 error 14 in 0[400000+6000]
13[2639]: segfault at 0 ip 0000000000403e5f sp 00007ffd98f61380 error 6 in 13[400000+6000]
13[2973]: segfault at 8 ip 00007fc1b74f97e8 sp 00007fff8cf53050 error 4 in ld-2.17.so[7fc1b74ee000+22000]
8[5509]: segfault at 1c ip 00000000004017b0 sp 00007ffd97cab560 error 6 in 8[400000+6000]
LustreError: 6420:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f0f2efc8: inode [0x200000403:0x9ed:0x0] mdc close failed: rc = -13
5[10969]: segfault at 8 ip 00007f5b716727e8 sp 00007fff51770ed0 error 4 in ld-2.17.so[7f5b71667000+22000]
1[12107]: segfault at 0 ip 0000000000403e5f sp 00007ffcaf260fd0 error 6 in 1[400000+6000]
18[12684]: segfault at 8 ip 00007fa37bb4a7e8 sp 00007ffe5fa06740 error 4 in ld-2.17.so[7fa37bb3f000+22000]
8[12804]: segfault at 0 ip (null) sp 00007ffc2b691f58 error 14 in 8[400000+6000]
17[12931]: segfault at 0 ip (null) sp 00007ffc1ef27cf8 error 14 in 17[400000+6000]
dd (14909) used greatest stack depth: 9600 bytes left
Lustre: 14170:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x3176:0x0] with magic=0xbd60bd0
Lustre: 14170:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
1[22189]: segfault at 8 ip 00007f3cbb9c97e8 sp 00007ffc45ffc980 error 4 in ld-2.17.so[7f3cbb9be000+22000]
15[23037]: segfault at 8 ip 00007fc5ce1db7e8 sp 00007fffef674440 error 4 in ld-2.17.so[7fc5ce1d0000+22000]
12[25319]: segfault at 8 ip 00007f5ec678f7e8 sp 00007ffe2c97c9c0 error 4 in ld-2.17.so[7f5ec6784000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD a122f067 PUD 840d1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks floppy ata_piix virtio_blk i2c_core serio_raw libata
CPU: 2 PID: 21293 Comm: ll_sa_13371 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802ac828010 ti: ffff8802cce94000 task.ti: ffff8802cce94000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802cce97d78 EFLAGS: 00010296
RAX: ffff8801ee9b4390 RBX: 0000000000000008 RCX: 0000000100260021
RDX: 0000000000000026 RSI: ffff8801ee9b46c0 RDI: 0000000000000008
RBP: ffff8802cce97d88 R08: ffff8802d91d5628 R09: 0000000000000000
R10: ffff8802d91d0358 R11: ffff8802d91d5628 R12: 0000000000000000
R13: ffff8802d061dc40 R14: ffff8801ee9b46c0 R15: ffff8802d91d5628
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000906ea000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15abbb1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa15aae00>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/5 is open, migrate only dentry
Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x200000401:0x4:0x0]/13 is open, migrate only dentry
Lustre: 31114:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/5 is open, migrate only dentry
Lustre: 31114:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 1 previous similar message
Lustre: 30390:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d8781940 x1832939896070656/t4294968153(0) o101->0826c2d5-6bc4-49c2-9c49-6fae1d6b9470@0@lo:463/0 lens 376/864 e 0 to 0 dl 1748027803 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
ls (28621) used greatest stack depth: 10144 bytes left
Lustre: 15189:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/7 is open, migrate only dentry
LustreError: 31026:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000404:0x2:0x0]/11 failed: rc = -2
LustreError: 30969:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration: rc = -1
LustreError: 30969:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000403:0x1:0x0]/5 failed: rc = -1
LustreError: 4201:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x67:0x0]: rc = -5
LustreError: 4201:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 2557:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000404:0x44:0x0]/14 failed: rc = -2
Lustre: 30356:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff880291474b40 x1832939899377280/t4294969557(0) o101->0826c2d5-6bc4-49c2-9c49-6fae1d6b9470@0@lo:484/0 lens 376/816 e 0 to 0 dl 1748027824 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 30663:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c07c2340 x1832939899853696/t4294969829(0) o101->0826c2d5-6bc4-49c2-9c49-6fae1d6b9470@0@lo:488/0 lens 376/840 e 0 to 0 dl 1748027828 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 28911:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x4a:0x0]: rc = -2
LustreError: 1483:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x240000403:0x4a:0x0] mdc close failed: rc = -2
LustreError: 1483:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 31190:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/9 is open, migrate only dentry
Lustre: 31190:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 1 previous similar message
LustreError: 6318:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '17' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 17' to finish migration: rc = -1
LustreError: 6318:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x2:0x0]/17 failed: rc = -1
LustreError: 30969:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000403:0x8e:0x0]/18 failed: rc = -2
LustreError: 30969:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 3 previous similar messages
Lustre: dir [0x280000403:0x8e:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 9781:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cf908008: inode [0x280000404:0x2c:0x0] mdc close failed: rc = -2
LustreError: 30126:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '14' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 14' to finish migration: rc = -1
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x1:0x0]/14 failed: rc = -1
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 31026:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 17' to finish migration: rc = -1
5[13741]: segfault at 8 ip 00007f3b58c9e7e8 sp 00007ffde1930ee0 error 4 in ld-2.17.so[7f3b58c93000+22000]
Lustre: 30126:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000401:0x6:0x0]/sleep is open, migrate only dentry
LustreError: 18515:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cf908008: inode [0x200000403:0x14:0x0] mdc close failed: rc = -2
LustreError: 22630:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x185:0x0]: rc = -5
LustreError: 22630:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 22630:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 22630:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 30948:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000404:0x192:0x0]/11 failed: rc = -2
LustreError: 30948:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 1 previous similar message
Lustre: 30369:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x1b2:0x0] with magic=0xbd60bd0
LustreError: 20250:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x240000404:0x185:0x0] : rc = -5
LustreError: 20250:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000404:0x185:0x0] error -5.
LustreError: 15191:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '17' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 17' to finish migration: rc = -1
LustreError: 23930:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x280000403:0x1a:0x0] mdc close failed: rc = -2
Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/18 is open, migrate only dentry
Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 3 previous similar messages
Lustre: dir [0x200000404:0x192:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 24624:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caa8c138: cannot apply new layout on [0x200000403:0x3f9:0x0] : rc = -5
LustreError: 24624:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x3f9:0x0] error -5.
LustreError: 30600:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x262:0x0] migrate mdt count mismatch 3 != 1
LustreError: 29496:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x240000404:0x185:0x0] : rc = -5
LustreError: 29496:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x185:0x0]: rc = -5
LustreError: 29496:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 27232:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x2ec:0x0]: rc = -5
LustreError: 27232:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
7[32356]: segfault at 8 ip 00007f44619f27e8 sp 00007ffcee2ce9a0 error 4 in ld-2.17.so[7f44619e7000+22000]
LustreError: 315:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x2df:0x0]: rc = -5
LustreError: 315:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 131:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 16 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 315:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 30704:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x203:0x0] with magic=0xbd60bd0
Lustre: 30704:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 30966:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000404:0x18a:0x0]/12 failed: rc = -2
LustreError: 30966:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 3 previous similar messages
Lustre: 30532:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x200:0x0] with magic=0xbd60bd0
Lustre: 30532:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 31190:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '13' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 13' to finish migration: rc = -1
LustreError: 5789:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x34b:0x0]: rc = -5
LustreError: 5789:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 5789:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 5789:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 3 previous similar messages
Lustre: 30966:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000400:0xf:0x0]/9 is open, migrate only dentry
Lustre: 30966:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 3 previous similar messages
Lustre: 30659:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x227:0x0] with magic=0xbd60bd0
Lustre: 30659:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000403:0x248:0x0]/0xa): rc = 0
LustreError: 13337:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x29f:0x0]: rc = -5
LustreError: 13337:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 8 previous similar messages
LustreError: 13337:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 13337:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 8 previous similar messages
Lustre: 28405:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x35a:0x0] with magic=0xbd60bd0
Lustre: 28405:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 15489:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x240000404:0x185:0x0] : rc = -5
17[12910]: segfault at 0 ip (null) sp 00007ffd3f23d008 error 14 in 17[400000+6000]
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000403:0x306:0x0]/0xa): rc = 0
LustreError: 6207:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000404:0x31c:0x0]/0xa): rc = 0
LustreError: 30692:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x691:0x0] migrate mdt count mismatch 3 != 2
LustreError: 30465:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x28e:0x0]: rc = -2
LustreError: 30465:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 16754:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x280000403:0x28e:0x0] mdc close failed: rc = -2
LustreError: 218:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x280000403:0x3ac:0x0] inode@0000000000000000: rc = -5
LustreError: 218:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 31026:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '4' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 4' to finish migration: rc = -1
LustreError: 28399:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x3f6:0x0]: rc = -5
LustreError: 28399:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 21 previous similar messages
LustreError: 28399:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 28399:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 21 previous similar messages
LustreError: 31114:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x5f5:0x0]/9 failed: rc = -2
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 11 previous similar messages
Lustre: dir [0x240000404:0x240:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
Lustre: 30969:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/13 is open, migrate only dentry
Lustre: 30969:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 7 previous similar messages
LustreError: 30642:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x5f5:0x0] migrate mdt count mismatch 2 != 1
LustreError: 6702:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '12' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 12' to finish migration: rc = -1
LustreError: 6702:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
LustreError: 306:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x240000404:0x185:0x0] : rc = -5
LustreError: 306:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000404:0x185:0x0] error -5.
Lustre: 30393:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x552:0x0] with magic=0xbd60bd0
Lustre: 30393:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 11 previous similar messages
Lustre: 28988:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x483:0x0] with magic=0xbd60bd0
Lustre: 28988:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
dd (17318) used greatest stack depth: 10016 bytes left
LustreError: 16921:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x280000404:0x43a:0x0] : rc = -5
LustreError: 16921:0:(lov_object.c:1350:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 10939:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x200000404:0x31c:0x0] mdc close failed: rc = -2
LustreError: 10939:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 19468:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x50e:0x0]: rc = -5
LustreError: 19468:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 21 previous similar messages
LustreError: 19468:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 19468:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 21 previous similar messages
LustreError: 22101:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x240000404:0x185:0x0] : rc = -5
Lustre: dir [0x240000403:0x612:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 16586:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x200000404:0x3d7:0x0] mdc close failed: rc = -2
LustreError: 315:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 3 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 2567:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 5' to finish migration: rc = -1
LustreError: 2567:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 7 previous similar messages
17[30918]: segfault at 8 ip 00007f8120f6d7e8 sp 00007ffcbb208e10 error 4 in ld-2.17.so[7f8120f62000+22000]
7[32544]: segfault at 8 ip 00007f75848047e8 sp 00007ffd08afbe30 error 4 in ld-2.17.so[7f75847f9000+22000]
14[1547]: segfault at 0 ip 0000000000403e5f sp 00007ffc8a7587a0 error 6 in 14[400000+6000]
Lustre: 30729:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x7f1:0x0] with magic=0xbd60bd0
Lustre: 30729:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 31 previous similar messages
LustreError: 10476:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x240000403:0x87f:0x0] dir layout mismatch:
LustreError: 10476:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 10476:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x240000400:0x36:0x0]
LustreError: 10476:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 10472:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 10472:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 7264:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x200000403:0x691:0x0] get parent: rc = -2
LustreError: 7262:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x200000403:0x691:0x0] mdc close failed: rc = -2
LustreError: 7262:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 15190:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x2:0x0]/11 failed: rc = -1
LustreError: 15190:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 30 previous similar messages
LustreError: 6970:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caa8c138: cannot apply new layout on [0x240000403:0x50e:0x0] : rc = -5
LustreError: 6970:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000403:0x50e:0x0] error -5.
LustreError: 6970:0:(vvp_io.c:1905:vvp_io_init()) Skipped 1 previous similar message
Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/19 is open, migrate only dentry
Lustre: 15191:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 40 previous similar messages
LustreError: 31:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 3 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
Lustre: dir [0x200000404:0x9fa:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 30639:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x2e6:0x0] migrate mdt count mismatch 3 != 2
LustreError: 28163:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x280000403:0xb91:0x0] dir layout mismatch:
LustreError: 28163:0:(llite_lib.c:1872:ll_update_lsm_md()) Skipped 1 previous similar message
LustreError: 28163:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 28163:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x280000400:0x31:0x0]
LustreError: 28163:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 9 previous similar messages
LustreError: 28163:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 28166:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 28166:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 22882:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000403:0x913:0x0] error -5.
15[29786]: segfault at 8 ip 00007f5b6e0db7e8 sp 00007ffe68c1e0f0 error 4 in ld-2.17.so[7f5b6e0d0000+22000]
Lustre: dir [0x240000404:0x934:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 19 previous similar messages
LustreError: 30128:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000404:0x583:0x0] error -5.
LustreError: 7368:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0xc2b:0x0]: rc = -5
LustreError: 7368:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 29 previous similar messages
LustreError: 7368:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 7368:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 29 previous similar messages
3[7801]: segfault at 8 ip 00007faf59bd47e8 sp 00007ffd8b263840 error 4 in ld-2.17.so[7faf59bc9000+22000]
LustreError: 4407:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cf908008: inode [0x280000403:0x7a3:0x0] mdc close failed: rc = -2
LustreError: 4407:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
LustreError: 15191:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '5' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 5' to finish migration: rc = -1
LustreError: 15191:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 10 previous similar messages
14[14255]: segfault at 8 ip 00007f2beba9a7e8 sp 00007ffd57d7be60 error 4 in ld-2.17.so[7f2beba8f000+22000]
14[15618]: segfault at 8 ip 00007f690b6997e8 sp 00007fffc8e1cc20 error 4 in ld-2.17.so[7f690b68e000+22000]
14[14855]: segfault at 8 ip 00007f4acc3c27e8 sp 00007ffc72d7f8e0 error 4 in ld-2.17.so[7f4acc3b7000+22000]
6[15704]: segfault at 8 ip 00007fcb83f817e8 sp 00007fff3ff40440 error 4 in ld-2.17.so[7fcb83f76000+22000]
5[15324]: segfault at 8 ip 00007f05f9b377e8 sp 00007ffe1c1458d0 error 4 in ld-2.17.so[7f05f9b2c000+22000]
LustreError: 6073:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caa8c138: cannot apply new layout on [0x280000403:0xc21:0x0] : rc = -5
LustreError: 6073:0:(lov_object.c:1350:lov_layout_change()) Skipped 6 previous similar messages
LustreError: 6073:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000403:0xc21:0x0] error -5.
LustreError: 28312:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0xbd6:0x0] migrate mdt count mismatch 2 != 1
11[29403]: segfault at 8 ip 00007ff8d37a57e8 sp 00007ffcf58e0690 error 4 in ld-2.17.so[7ff8d379a000+22000]
6[3126]: segfault at 8 ip 00007f91071c47e8 sp 00007ffdb62870e0 error 4 in ld-2.17.so[7f91071b9000+22000]
Lustre: 30376:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0xc27:0x0] with magic=0xbd60bd0
Lustre: 30376:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 15 previous similar messages
LustreError: 316:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x280000403:0xcb2:0x0] inode@0000000000000000: rc = -5
LustreError: 31702:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x6de:0x0] migrate mdt count mismatch 2 != 1
LustreError: 31702:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
16[14976]: segfault at 8 ip 00007fad7fffd7e8 sp 00007ffefa6ac780 error 4 in ld-2.17.so[7fad7fff2000+22000]
LustreError: 31:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x280000403:0xcb2:0x0] inode@0000000000000000: rc = -5
Lustre: dir [0x280000404:0xc89:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 14238:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000404:0xdb2:0x0] error -5.
LustreError: 28022:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x97b:0x0]: rc = -2
LustreError: 28022:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 8584:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cf908008: inode [0x200000404:0x97b:0x0] mdc close failed: rc = -2
LustreError: 8584:0:(file.c:247:ll_close_inode_openhandle()) Skipped 4 previous similar messages
13[24265]: segfault at 8 ip 00007f30dc22c7e8 sp 00007ffdbb0b40d0 error 4 in ld-2.17.so[7f30dc221000+22000]
LustreError: 30318:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0xf6b:0x0] migrate mdt count mismatch 2 != 3
LustreError: 30318:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
LustreError: 127:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x280000403:0xcb2:0x0] inode@0000000000000000: rc = -5
Lustre: dir [0x240000404:0xca3:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 615:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0xc1b:0x0]: rc = -2
LustreError: 615:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 2327:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caa8c138: cannot apply new layout on [0x280000403:0xffc:0x0] : rc = -5
LustreError: 2327:0:(lov_object.c:1350:lov_layout_change()) Skipped 10 previous similar messages
LustreError: 30393:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x1083:0x0] migrate mdt count mismatch 2 != 3
LustreError: 316:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 11 [0x240000403:0xbb7:0x0] inode@0000000000000000: rc = -5
5[10360]: segfault at 8 ip 00007fee381447e8 sp 00007ffd20327a20 error 4 in ld-2.17.so[7fee38139000+22000]
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x1488:0x0]/3 failed: rc = -2
LustreError: 30126:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 53 previous similar messages
Lustre: 6207:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x240000403:0x1:0x0]/10 is open, migrate only dentry
Lustre: 6207:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 47 previous similar messages
LustreError: 15386:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1499:0x0] error -5.
LustreError: 15386:0:(vvp_io.c:1905:vvp_io_init()) Skipped 3 previous similar messages
Lustre: 23567:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x1181:0x0] with magic=0xbd60bd0
Lustre: 23567:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 57 previous similar messages
LustreError: 23080:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 16 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 23080:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 9 previous similar messages
Lustre: mdt00_000: service thread pid 15170 was inactive for 72.119 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 15170, comm: mdt00_000 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xabc/0xfd0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xb00 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x3c4/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_link+0x8d4/0xf40 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_016: service thread pid 28488 was inactive for 72.267 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 28488, comm: mdt00_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf6/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_033: service thread pid 30351 was inactive for 72.195 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 30351, comm: mdt00_033 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xc4a/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: dir [0x280000403:0x122c:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 21 previous similar messages
LustreError: 52:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x13be:0x0]: rc = -5
LustreError: 52:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 119 previous similar messages
LustreError: 52:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 52:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 119 previous similar messages
14[4917]: segfault at 8 ip 00007f05ee3267e8 sp 00007fff32dc3750 error 4 in ld-2.17.so[7f05ee31b000+22000]
15[3218]: segfault at 8 ip 00007f1cf8b0c7e8 sp 00007ffc495ce050 error 4 in ld-2.17.so[7f1cf8b01000+22000]
LustreError: 6330:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration: rc = -1
LustreError: 6330:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 15 previous similar messages
LustreError: 14993:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802a02d4b40/0xa9fda64c8cb13e53 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x143e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa9fda64c8cb13e30 expref: 777 pid: 30532 timeout: 784 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802caa8c138: operation mds_getxattr to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802caa8c138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802caa8c138: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 8111:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 29524:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8802caa8c138: [0x200000402:0x4a:0x0] lock enqueue fails: rc = -5
LustreError: 29524:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
Lustre: mdt00_000: service thread pid 15170 completed after 100.196s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 28488 completed after 99.565s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 8812:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8802caa8c138: namespace resource [0x200000403:0x143e:0x0].0x0 (ffff8802d53e16c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: mdt00_033: service thread pid 30351 completed after 98.244s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8802caa8c138: Connection restored to 192.168.123.72@tcp (at 0@lo)
LustreError: 30643:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000405:0x114:0x0] migrate mdt count mismatch 1 != 2
LustreError: 30643:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 2 previous similar messages
2[9222]: segfault at 0 ip (null) sp 00007ffc5f69f2e8 error 14 in 2[400000+6000]
LustreError: 8202:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cf908008: inode [0x200000405:0x23a:0x0] mdc close failed: rc = -2
LustreError: 8202:0:(file.c:247:ll_close_inode_openhandle()) Skipped 28 previous similar messages
Lustre: dir [0x280000404:0x1553:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
LustreError: 30636:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x1c7f:0x0] ACL: rc = -2
LustreError: 131:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 12 [0x200000405:0x3eb:0x0] inode@0000000000000000: rc = -5
LustreError: 131:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 5 previous similar messages
LustreError: 6167:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802caa8c138: cannot apply new layout on [0x200000403:0x1499:0x0] : rc = -5
LustreError: 6167:0:(lov_object.c:1350:lov_layout_change()) Skipped 8 previous similar messages
LustreError: 28978:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x1b61:0x0] doesn't exist!: rc = -14
LustreError: 30659:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x1c40:0x0] ACL: rc = -2
LustreError: 2567:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
Lustre: 28978:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x1782:0x0] with magic=0xbd60bd0
Lustre: 28978:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 123 previous similar messages
LustreError: 30659:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x1a4c:0x0] migrate mdt count mismatch 2 != 1
LustreError: 30659:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 2 previous similar messages
12[32078]: segfault at 0 ip 00007fd41952959a sp 00007fff9b519ce0 error 4 in ld-2.17.so[7fd419527000+22000]
LustreError: 28235:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000404:0x1555:0x0] error -5.
LustreError: 28235:0:(vvp_io.c:1905:vvp_io_init()) Skipped 1 previous similar message
10[9789]: segfault at 8 ip 00007f04e94aa7e8 sp 00007ffd8f2a4d50 error 4 in ld-2.17.so[7f04e949f000+22000]
Lustre: mdt_io00_000: service thread pid 15189 was inactive for 40.005 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 15189, comm: mdt_io00_000 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0x57/0xf0 [mdt]
[<0>] mdt_reint_migrate+0x1832/0x24b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 30966, comm: mdt_io00_005 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xabc/0xfd0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xb00 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x3c4/0x470 [mdt]
[<0>] mdt_rename_lock+0xd9/0x360 [mdt]
[<0>] mdt_reint_migrate+0x87e/0x24b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 15191, comm: mdt_io00_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_rename_lock+0xd9/0x360 [mdt]
[<0>] mdt_reint_migrate+0x87e/0x24b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io00_018: service thread pid 6702 was inactive for 40.087 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io00_010: service thread pid 31186 was inactive for 40.083 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: mdt_io00_021: service thread pid 959 was inactive for 40.014 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt_io00_003: service thread pid 30126 was inactive for 40.046 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
LustreError: 14993:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88009e659e40/0xa9fda64c8ce81432 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1aa6:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa9fda64c8ce813ec expref: 473 pid: 15178 timeout: 1214 lvb_type: 0
Lustre: mdt_io00_000: service thread pid 15189 completed after 102.001s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 14984:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748028870 with bad export cookie 12249129409220666952
Lustre: lustre-MDT0001-mdc-ffff8802cf908008: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff8802cf908008: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 13944:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff8802cf908008: [0x240000400:0x76:0x0] lock enqueue fails: rc = -108
LustreError: 13944:0:(mdc_request.c:1469:mdc_read_page()) Skipped 3 previous similar messages
Lustre: dir [0x240000404:0x17bc:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 29316:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0001-mdc-ffff8802cf908008: Connection restored to 192.168.123.72@tcp (at 0@lo)
Lustre: mdt_io00_005: service thread pid 30966 completed after 101.752s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_002: service thread pid 15191 completed after 101.753s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_018: service thread pid 6702 completed after 101.244s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_020: service thread pid 13213 completed after 101.476s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_007: service thread pid 31026 completed after 101.391s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 2567:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000403:0x211b:0x0]/14 failed: rc = -2
LustreError: 2567:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 79 previous similar messages
Lustre: mdt_io00_013: service thread pid 2567 completed after 101.281s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_012: service thread pid 2557 completed after 102.313s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_010: service thread pid 31186 completed after 101.175s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_017: service thread pid 6330 completed after 100.484s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_021: service thread pid 959 completed after 99.000s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 15190:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/19 is open, migrate only dentry
Lustre: 15190:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 69 previous similar messages
Lustre: mdt_io00_001: service thread pid 15190 completed after 98.618s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_019: service thread pid 7119 completed after 98.885s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_016: service thread pid 6318 completed after 98.870s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_003: service thread pid 30126 completed after 95.779s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_022: service thread pid 1415 completed after 95.029s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_009: service thread pid 31143 completed after 94.569s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_008: service thread pid 31114 completed after 94.019s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
7[4913]: segfault at 8 ip 00007f5483ec87e8 sp 00007fff429e9660 error 4 in ld-2.17.so[7f5483ebd000+22000]
7[5159]: segfault at 8 ip 00007fa0a383a7e8 sp 00007ffe105c72b0 error 4 in ld-2.17.so[7fa0a382f000+22000]
LustreError: 317:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 7 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 317:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 12 previous similar messages
0[11318]: segfault at 8 ip 00007f25ee0f67e8 sp 00007ffea9079a30 error 4 in ld-2.17.so[7f25ee0eb000+22000]
LustreError: 17674:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x1997:0x0]: rc = -5
LustreError: 17674:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 179 previous similar messages
LustreError: 17674:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 17674:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 179 previous similar messages
LustreError: 959:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '1' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 1' to finish migration: rc = -1
LustreError: 959:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 27 previous similar messages
LustreError: 15189:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 4204:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x1c48:0x0] migrate mdt count mismatch 3 != 1
LustreError: 4204:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
LustreError: 10038:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000403:0x199f:0x0] error -5.
LustreError: 10038:0:(vvp_io.c:1905:vvp_io_init()) Skipped 3 previous similar messages
LustreError: 31186:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
6[7571]: segfault at 8 ip 00007ff94f17a7e8 sp 00007ffd51871f20 error 4 in ld-2.17.so[7ff94f16f000+22000]
13[12002]: segfault at 8 ip 00007f1ac9eab7e8 sp 00007ffdf34ac010 error 4 in ld-2.17.so[7f1ac9ea0000+22000]
LustreError: 30659:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000405:0x339:0x0] ACL: rc = -2
LustreError: 15174:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x1aa4:0x0]: rc = -2
LustreError: 15174:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
9[24075]: segfault at 8 ip 00007f59e80ce7e8 sp 00007ffc61d734d0 error 4 in ld-2.17.so[7f59e80c3000+22000]
LustreError: 25889:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x280000403:0x206a:0x0] mdc close failed: rc = -2
LustreError: 25889:0:(file.c:247:ll_close_inode_openhandle()) Skipped 43 previous similar messages
LustreError: 5520:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x200000403:0x2355:0x0] get parent: rc = -2
LustreError: 5520:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) Skipped 4 previous similar messages
LustreError: 16719:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x280000404:0x2196:0x0] : rc = -5
LustreError: 16719:0:(lov_object.c:1350:lov_layout_change()) Skipped 12 previous similar messages
Lustre: 15170:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2852:0x0] with magic=0xbd60bd0
Lustre: 15170:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 163 previous similar messages
Lustre: 15189:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0002: [0x280000403:0x1:0x0]/18 is open, migrate only dentry
LustreError: 7528:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X1000.
LustreError: 28930:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0002: [0x280000404:0x251b:0x0] doesn't exist!: rc = -14
17[14120]: segfault at 8 ip 00007fa2ec62c7e8 sp 00007ffd2d515650 error 4 in ld-2.17.so[7fa2ec621000+22000]
11[20299]: segfault at b6 ip 00000000000000b6 sp 00007ffe2bd555a8 error 14 in 11[400000+6000]
LustreError: 6330:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 1927:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x280000404:0x2386:0x0] get parent: rc = -2
LustreError: 1927:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) Skipped 1 previous similar message
3[26749]: segfault at 8 ip 00007fed37d577e8 sp 00007ffed0421560 error 4 in ld-2.17.so[7fed37d4c000+22000]
0[10855]: segfault at 8 ip 00007f33017087e8 sp 00007ffd1fa4e8b0 error 4 in ld-2.17.so[7f33016fd000+22000]
11[21694]: segfault at 8 ip 00007f8e7fa967e8 sp 00007ffe36b52240 error 4 in ld-2.17.so[7f8e7fa8b000+22000]
LustreError: 14993:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802d34643c0/0xa9fda64c8d2b4f9c lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x3204:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa9fda64c8d2b4f80 expref: 604 pid: 30318 timeout: 1814 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8802cf908008: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802cf908008: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802cf908008: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 24001:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 24001:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 14 previous similar messages
LustreError: 24210:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8802cf908008: [0x200000400:0x9e:0x0] lock enqueue fails: rc = -108
LustreError: 24210:0:(mdc_request.c:1469:mdc_read_page()) Skipped 6 previous similar messages
Lustre: dir [0x200000403:0x3135:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 51 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8802cf908008: Connection restored to 192.168.123.72@tcp (at 0@lo)
LustreError: 30457:0:(mdd_orphans.c:261:mdd_orphan_delete()) lustre-MDD0001: could not delete orphan object [0x240000404:0x2834:0x0]: rc = -2
LustreError: 30457:0:(mdd_object.c:3955:mdd_close()) lustre-MDD0001: unable to delete [0x240000404:0x2834:0x0] from orphan list: rc = -2
LustreError: 959:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000404:0x29ad:0x0]/1 failed: rc = -116
LustreError: 959:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 75 previous similar messages
Lustre: 2567:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/16 is open, migrate only dentry
Lustre: 2567:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 87 previous similar messages
LustreError: 5826:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x2aa2:0x0] ACL: rc = -2
14[9212]: segfault at 8 ip 00007fbfce52e7e8 sp 00007fff1107e530 error 4 in ld-2.17.so[7fbfce523000+22000]
LustreError: 31751:0:(osd_io.c:901:osd_grow_blocksize()) object [0x340000400:0x4b7:0x0]: change block size4096 -> 1048576 error rc = -95
0[17599]: segfault at 8 ip 00007fd0691b77e8 sp 00007ffe9ef55ba0 error 4 in ld-2.17.so[7fd0691ac000+22000]
LustreError: 5730:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x29b3:0x0] ACL: rc = -2
LustreError: 5854:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0x1c66:0x0] migrate mdt count mismatch 3 != 1
LustreError: 5854:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 13 previous similar messages
LustreError: 31988:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0x1a27:0x0]: rc = -5
LustreError: 31988:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 239 previous similar messages
LustreError: 31988:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 31988:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 239 previous similar messages
LustreError: 31186:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 0' to finish migration: rc = -1
LustreError: 31186:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 27 previous similar messages
LustreError: 661:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 10 [0x200000406:0x336:0x0] inode@0000000000000000: rc = -5
LustreError: 661:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 9 previous similar messages
18[26585]: segfault at 8 ip 00007f63f5fe47e8 sp 00007ffe2a554a40 error 4 in ld-2.17.so[7f63f5fd9000+22000]
LustreError: 30770:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000403:0x31b3:0x0] error -5.
LustreError: 30770:0:(vvp_io.c:1905:vvp_io_init()) Skipped 8 previous similar messages
0[23752]: segfault at 8 ip 00007f0cd85ae7e8 sp 00007ffd514fd550 error 4 in ld-2.17.so[7f0cd85a3000+22000]
LustreError: 15439:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802caa8c138: inode [0x200000406:0x60f:0x0] mdc close failed: rc = -2
LustreError: 15439:0:(file.c:247:ll_close_inode_openhandle()) Skipped 54 previous similar messages
13[1470]: segfault at 8 ip 00007f46a14317e8 sp 00007ffd513bf1a0 error 4 in ld-2.17.so[7f46a1426000+22000]
19[7684]: segfault at 8 ip 00007f36d0d637e8 sp 00007ffce71bcda0 error 4 in ld-2.17.so[7f36d0d58000+22000]
Lustre: 30642:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000406:0x515:0x0] with magic=0xbd60bd0
Lustre: 30642:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 97 previous similar messages
LustreError: 5733:0:(lov_object.c:1350:lov_layout_change()) lustre-clilov-ffff8802cf908008: cannot apply new layout on [0x280000404:0x31d9:0x0] : rc = -5
LustreError: 5733:0:(lov_object.c:1350:lov_layout_change()) Skipped 15 previous similar messages
19[28414]: segfault at 0 ip 0000000000403e5f sp 00007fff0e472080 error 6 in 19[400000+6000]
5[29069]: segfault at 8 ip 00007fb27aeb97e8 sp 00007ffc6cc39860 error 4 in ld-2.17.so[7fb27aeae000+22000]
LustreError: 31702:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '17' present, but FID [0x200000406:0x266:0x0] is invalid
LustreError: 28484:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '17' present, but FID [0x200000406:0x266:0x0] is invalid
LustreError: 19216:0:(llite_lib.c:1872:ll_update_lsm_md()) lustre: [0x200000406:0x9a5:0x0] dir layout mismatch:
LustreError: 19216:0:(llite_lib.c:1872:ll_update_lsm_md()) Skipped 1 previous similar message
LustreError: 19216:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 19216:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0xd5:0x0]
LustreError: 19216:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) Skipped 13 previous similar messages
LustreError: 19216:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:2000003 pool=
LustreError: 10785:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 10785:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:2000003 pool=
LustreError: 14993:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802d51da5c0/0xa9fda64c8d6383cf lrc: 3/0,0 mode: PR/PR res: [0x280000404:0x2ff1:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa9fda64c8d6383b3 expref: 543 pid: 31433 timeout: 2303 lvb_type: 0
LustreError: 25265:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748029959 with bad export cookie 12249129409220667197
LustreError: lustre-MDT0002-mdc-ffff8802caa8c138: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802caa8c138: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8802caa8c138: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 29393:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0002-mdc-ffff8802caa8c138: Connection restored to 192.168.123.72@tcp (at 0@lo)
12[2501]: segfault at 8 ip 00007fee354eb7e8 sp 00007ffc6c728760 error 4 in ld-2.17.so[7fee354e0000+22000]
LustreError: 14471:0:(mdt_open.c:1703:mdt_reint_open()) lustre-MDT0000: name '17' present, but FID [0x200000406:0x266:0x0] is invalid
LustreError: 14993:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802a02d0400/0xa9fda64c8d693159 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x3730:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa9fda64c8d69312f expref: 598 pid: 30633 timeout: 2428 lvb_type: 0
LustreError: lustre-MDT0001-mdc-ffff8802caa8c138: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 11785:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1748030084 with bad export cookie 12249129409220667190
Lustre: lustre-MDT0001-mdc-ffff8802caa8c138: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff8802caa8c138: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 9614:0:(file.c:6187:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x3730:0x0] error: rc = -5
LustreError: 9614:0:(file.c:6187:ll_inode_revalidate_fini()) Skipped 21 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff8802caa8c138: Connection restored to 192.168.123.72@tcp (at 0@lo)
Lustre: 2557:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/13 is open, migrate only dentry
Lustre: 2557:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 78 previous similar messages
LustreError: 31143:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000404:0x3336:0x0]/16 failed: rc = -2
LustreError: 31143:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 71 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2c3d0a067 PUD 2eb37a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata
CPU: 12 PID: 28326 Comm: ll_sa_28292 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802eb2d5c40 ti: ffff8802ab22c000 task.ti: ffff8802ab22c000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802ab22fd78 EFLAGS: 00010296
RAX: ffff88029f394390 RBX: 0000000000000008 RCX: 000000010026001e
RDX: 0000000000000026 RSI: ffff88029f3946c0 RDI: 0000000000000008
RBP: ffff8802ab22fd88 R08: ffff8800a74f0008 R09: 0000000000000000
R10: ffff8800a74f0358 R11: ffff8800a74f0008 R12: 0000000000000000
R13: ffff8802ea9a5c40 R14: ffff88029f3946c0 R15: ffff8800a74f0008
FS: 0000000000000000(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002c3f26000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1110111>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa110f360>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 14169:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802eaa31e40 x1832383867562880/t4294967671(0) o101->b2e44554-0b11-4228-809b-933680fca1f4@0@lo:167/0 lens 376/840 e 0 to 0 dl 1747497497 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 9269:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9269:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9269:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9269:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9269:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9269:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9268:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9268:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9268:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 9268:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9268:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9268:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9268:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 20547:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000402:0x1d6:0x0] mdc close failed: rc = -13
Lustre: 9270:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9270:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 9270:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9270:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9270:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9270:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9270:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9270:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9270:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9270:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9270:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9270:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
LustreError: 21005:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000401:0x21e:0x0] mdc close failed: rc = -13
Lustre: 9268:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9268:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 9268:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9268:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9268:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9268:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9268:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 9268:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9268:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9268:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9268:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9268:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 5 previous similar messages
7[25390]: segfault at 8 ip 00007f72ff5b87e8 sp 00007ffd6fa814d0 error 4 in ld-2.17.so[7f72ff5ad000+22000]
LustreError: 25781:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000402:0x365:0x0] mdc close failed: rc = -13
Lustre: 9269:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9269:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 9269:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9269:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 9269:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9269:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 9269:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9269:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 9269:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9269:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 9269:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9269:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 17 previous similar messages
traps: 8[29990] trap invalid opcode ip:40543a sp:7ffc06ecc9c8 error:0 in 8[400000+6000]
17[30437]: segfault at 8 ip 00007f8a160d77e8 sp 00007ffdb9ffcad0 error 4 in ld-2.17.so[7f8a160cc000+22000]
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 540, rollback = 7
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 33 previous similar messages
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/540/0, punch: 0/0/0, quota 4/150/0
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 33 previous similar messages
8[1305]: segfault at 0 ip 0000000000404470 sp 00007ffee8732be8 error 6 in 10[400000+6000]
8[1926]: segfault at 8 ip 00007f7aaf7007e8 sp 00007ffda0763d20 error 4 in ld-2.17.so[7f7aaf6f5000+22000]
Lustre: 16371:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x606:0x0] with magic=0xbd60bd0
13[4255]: segfault at 8 ip 00007fc14da047e8 sp 00007ffd39fc5fb0 error 4 in ld-2.17.so[7fc14d9f9000+22000]
5[4346]: segfault at 0 ip 0000000000403e5f sp 00007ffd32cd2270 error 6 in 5[400000+6000]
LustreError: 4346:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9deb7e8: inode [0x200000401:0x698:0x0] mdc close failed: rc = -13
13[7661]: segfault at 8 ip 00007f9ef2b0f7e8 sp 00007ffe1cbe7510 error 4 in ld-2.17.so[7f9ef2b04000+22000]
Lustre: 8538:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x85e:0x0] with magic=0xbd60bd0
Lustre: 8538:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 43 previous similar messages
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 43 previous similar messages
2[14925]: segfault at 8 ip 00007f76f1f277e8 sp 00007ffed7589f40 error 4 in ld-2.17.so[7f76f1f1c000+22000]
3[17326]: segfault at 406000 ip 0000000000406000 sp 00007ffccb9d6fb8 error 14 in 3[606000+1000]
14[18926]: segfault at 8 ip 00007f090c1ab7e8 sp 00007ffeb3c62f70 error 4 in ld-2.17.so[7f090c1a0000+22000]
traps: 9[20910] trap invalid opcode ip:4058fa sp:7fffccbd0a58 error:0 in 12[400000+6000]
3[26731]: segfault at 8 ip 00007fd3c63b87e8 sp 00007ffdabd7a9f0 error 4 in ld-2.17.so[7fd3c63ad000+22000]
Lustre: 14848:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0xe20:0x0] with magic=0xbd60bd0
Lustre: 14848:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 32624:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000401:0xfab:0x0] mdc close failed: rc = -13
Lustre: 14383:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 14383:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 151 previous similar messages
Lustre: 14383:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14383:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 151 previous similar messages
Lustre: 14383:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14383:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 151 previous similar messages
Lustre: 14383:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 14383:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 151 previous similar messages
Lustre: 14383:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14383:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 151 previous similar messages
Lustre: 14383:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14383:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 151 previous similar messages
Lustre: 8538:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x10e4:0x0] with magic=0xbd60bd0
Lustre: 8538:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 9100:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9deb7e8: inode [0x200000401:0x1262:0x0] mdc close failed: rc = -13
LustreError: 9100:0:(file.c:247:ll_close_inode_openhandle()) Skipped 3 previous similar messages
7[12511]: segfault at 8 ip 00007f3c7b8a17e8 sp 00007ffc422ebc60 error 4 in ld-2.17.so[7f3c7b896000+22000]
Lustre: 11927:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x13dd:0x0] with magic=0xbd60bd0
Lustre: 11927:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
5[17923]: segfault at 8 ip 00007f4b301837e8 sp 00007fff0913f810 error 4 in ld-2.17.so[7f4b30178000+22000]
5[19198]: segfault at 8 ip 00007f9ee77bd7e8 sp 00007ffe334ce5b0 error 4 in ld-2.17.so[7f9ee77b2000+22000]
2[24304]: segfault at 8 ip 00007f0fae5347e8 sp 00007fff9b643450 error 4 in ld-2.17.so[7f0fae529000+22000]
1[28530]: segfault at 8 ip 00007fe4360817e8 sp 00007ffc60374c30 error 4 in ld-2.17.so[7fe436076000+22000]
17[28719]: segfault at 8 ip 00007ff6c246d7e8 sp 00007ffdc78b2f30 error 4 in ld-2.17.so[7ff6c2462000+22000]
17[28779]: segfault at 8 ip 00007f0e7a1fe7e8 sp 00007fff47d8c9c0 error 4 in ld-2.17.so[7f0e7a1f3000+22000]
5[31151]: segfault at 8 ip 00007f495b1bf2fc sp 00007ffe063b0ac0 error 4 in ld-2.17.so[7f495b1b2000+22000]
LustreError: 1320:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000402:0x1a4e:0x0] mdc close failed: rc = -13
LustreError: 1320:0:(file.c:247:ll_close_inode_openhandle()) Skipped 2 previous similar messages
10[4017]: segfault at 0 ip 00000000004043f0 sp 00007ffed44f2798 error 6 in 10[400000+6000]
Lustre: 5126:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1c1b:0x0] with magic=0xbd60bd0
Lustre: 5126:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
14[9476]: segfault at 8 ip 00007fa09a6107e8 sp 00007fff59dddc20 error 4 in ld-2.17.so[7fa09a605000+22000]
LustreError: 9713:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9deb7e8: inode [0x200000402:0x1cdf:0x0] mdc close failed: rc = -13
Lustre: 16367:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1db0:0x0] with magic=0xbd60bd0
Lustre: 16367:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
17[11621]: segfault at 406000 ip 0000000000406000 sp 00007ffebb840db8 error 14 in 17[606000+1000]
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 20714:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 279 previous similar messages
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20714:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 279 previous similar messages
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20714:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 279 previous similar messages
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 20714:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 279 previous similar messages
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20714:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 279 previous similar messages
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20714:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 279 previous similar messages
3[29568]: segfault at 8 ip 00007f3d8dda57e8 sp 00007ffe4d6ca4d0 error 4 in ld-2.17.so[7f3d8dd9a000+22000]
5[3341]: segfault at 8 ip 00007f6892f537e8 sp 00007ffe18fe7920 error 4 in ld-2.17.so[7f6892f48000+22000]
16[5083]: segfault at 0 ip 0000000000403e5f sp 00007ffecd4a71d0 error 6 in 16 (deleted)[400000+6000]
LustreError: 6776:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000402:0x2656:0x0] mdc close failed: rc = -13
10[8294]: segfault at 8 ip 00007f598e5257e8 sp 00007ffd0e7e4240 error 4 in ld-2.17.so[7f598e51a000+22000]
16[9919]: segfault at 8 ip 00007fa5325197e8 sp 00007ffd94ff7b00 error 4 in ld-2.17.so[7fa53250e000+22000]
Lustre: 14169:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x2a3c:0x0] with magic=0xbd60bd0
Lustre: 14169:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
5[17093]: segfault at 8 ip 00007fcfd278b7e8 sp 00007ffeee894cd0 error 4 in ld-2.17.so[7fcfd2780000+22000]
11[23234]: segfault at 8 ip 00007f06489b97e8 sp 00007ffe846573c0 error 4 in ld-2.17.so[7f06489ae000+22000]
5[24208]: segfault at 8 ip 00007fd3e6d307e8 sp 00007fffde944cd0 error 4 in ld-2.17.so[7fd3e6d25000+22000]
5[31558]: segfault at 0 ip 00000000004043f0 sp 00007ffdadc03568 error 6 in 5[400000+6000]
9[32584]: segfault at 8 ip 00007f49990577e8 sp 00007ffe0bb14170 error 4 in ld-2.17.so[7f499904c000+22000]
6[6445]: segfault at 406000 ip 0000000000406000 sp 00007fffc19ce648 error 14 in 6[606000+1000]
2[6842]: segfault at 0 ip 0000000000403e5f sp 00007fff29a16480 error 6 in 2[400000+6000]
0[7932]: segfault at 8 ip 00007ff6a3acc7e8 sp 00007ffc1c33c2e0 error 4 in ld-2.17.so[7ff6a3ac1000+22000]
2[8579]: segfault at 0 ip 0000000000403e5f sp 00007ffef3334040 error 6 in 12[400000+6000]
LustreError: 8982:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9deb7e8: inode [0x200000401:0x3288:0x0] mdc close failed: rc = -13
LustreError: 8982:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
18[15974]: segfault at 8 ip 00007fa908a6d7e8 sp 00007ffc2eca0460 error 4 in ld-2.17.so[7fa908a62000+22000]
Lustre: mdt_io00_002: service thread pid 8556 was inactive for 40.088 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 8556, comm: mdt_io00_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
Lustre: mdt_out00_000: service thread pid 8543 was inactive for 40.079 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_check_lock+0xec/0x3c0 [mdt]
[<0>] mdt_reint_rename+0x148f/0x3950 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 8545, comm: mdt_out00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1844/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 13821, comm: mdt_out00_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x369/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1844/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0002-osc-ffff8800a9deb7e8: disconnect after 21s idle
Lustre: lustre-OST0003-osc-ffff8800a9deb7e8: disconnect after 21s idle
Lustre: Skipped 1 previous similar message
LustreError: 8318:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802e0ca07c0/0xfa2dee2eafd111a1 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x35d4:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xfa2dee2eafd11035 expref: 1198 pid: 8545 timeout: 1034 lvb_type: 0
LustreError: 8556:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802c4e91940 x1832384008059264/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: mdt_io00_002: service thread pid 8556 completed after 100.129s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 11372 completed after 100.070s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 24846:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a9dbb7e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a56c8f40/0xfa2dee2eafd1151a lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x35d4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xfa2dee2eafd1150c expref: 446 pid: 24846 timeout: 0 lvb_type: 0
Lustre: mdt00_012: service thread pid 24846 completed after 100.036s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a9deb7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800a9deb7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt_out00_001: service thread pid 8545 completed after 100.108s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_002: service thread pid 13821 completed after 100.144s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_000: service thread pid 8543 completed after 100.130s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a9deb7e8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 17918:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x35d4:0x0] error: rc = -5
LustreError: 18153:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8800a9deb7e8: [0x200000401:0x1:0x0] lock enqueue fails: rc = -5
LustreError: 18348:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a9deb7e8: namespace resource [0x200000401:0x1:0x0].0x0 (ffff8802d2c7f240) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff8800a9deb7e8: Connection restored to 192.168.123.32@tcp (at 0@lo)
1[26481]: segfault at 8 ip 00007f547c18a7e8 sp 00007ffefaa51320 error 4 in ld-2.17.so[7f547c17f000+22000]
17[27247]: segfault at 8 ip 00007fde654187e8 sp 00007ffdc451b020 error 4 in ld-2.17.so[7fde6540d000+22000]
0[29976]: segfault at 8 ip 00007f0d294497e8 sp 00007ffe4bcaf920 error 4 in ld-2.17.so[7f0d2943e000+22000]
13[8321]: segfault at 8 ip 00007f719a4c77e8 sp 00007ffce90343d0 error 4 in ld-2.17.so[7f719a4bc000+22000]
LustreError: 16655:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9ea2e98: inode [0x200000402:0x3ebe:0x0] mdc close failed: rc = -13
LustreError: 16655:0:(file.c:247:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 570, rollback = 7
Lustre: 14468:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 313 previous similar messages
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14468:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 313 previous similar messages
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14468:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 313 previous similar messages
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/570/0, punch: 0/0/0, quota 1/3/0
Lustre: 14468:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 313 previous similar messages
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14468:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 313 previous similar messages
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14468:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 313 previous similar messages
Lustre: 16371:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xb7c:0x0] with magic=0xbd60bd0
Lustre: 16371:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
4[25238]: segfault at 8 ip 00007f154e60f7e8 sp 00007ffee3179180 error 4 in ld-2.17.so[7f154e604000+22000]
16[30852]: segfault at 8 ip 00007f40583567e8 sp 00007ffcc7f3abb0 error 4 in ld-2.17.so[7f405834b000+22000]
13[30854]: segfault at 8 ip 00007ff7e338c7e8 sp 00007ffd63aa6bb0 error 4 in ld-2.17.so[7ff7e3381000+22000]
13[30918]: segfault at 8 ip 00007f1a84bcb7e8 sp 00007ffdbbe8e0e0 error 4 in ld-2.17.so[7f1a84bc0000+22000]
mkdir (31663) used greatest stack depth: 9968 bytes left
8[705]: segfault at 0 ip 0000000000401d60 sp 00007ffc8b116c88 error 6 in 8[400000+6000]
5[1262]: segfault at 8 ip 00007fe861bd57e8 sp 00007ffc9f81d130 error 4 in ld-2.17.so[7fe861bca000+22000]
17[11181]: segfault at 8 ip 00007fdbef8f67e8 sp 00007fffcbc5fa80 error 4 in ld-2.17.so[7fdbef8eb000+22000]
19[12714]: segfault at 8 ip 00007f71619717e8 sp 00007ffda6008860 error 4 in ld-2.17.so[7f7161966000+22000]
18[15781]: segfault at 8 ip 00007faeefea27e8 sp 00007ffcacb831f0 error 4 in ld-2.17.so[7faeefe97000+22000]
14[23800]: segfault at 8 ip 00007fca6629f7e8 sp 00007ffd55d5d3a0 error 4 in ld-2.17.so[7fca66294000+22000]
13[27096]: segfault at 8 ip 00007fce6b2ff7e8 sp 00007ffe44747040 error 4 in ld-2.17.so[7fce6b2f4000+22000]
7[664]: segfault at 8 ip 00007fa346aee7e8 sp 00007ffc5855f460 error 4 in ld-2.17.so[7fa346ae3000+22000]
12[5405]: segfault at 8 ip 00007fdfad6667e8 sp 00007ffd23207500 error 4 in ld-2.17.so[7fdfad65b000+22000]
12[6134]: segfault at 0 ip 0000000000403e5f sp 00007ffd27352250 error 6 in 12[400000+6000]
Lustre: 14848:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1dfa:0x0] with magic=0xbd60bd0
Lustre: 14848:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 5 previous similar messages
18[14016]: segfault at 8 ip 00007ff7341657e8 sp 00007ffcdeca4a50 error 4 in ld-2.17.so[7ff73415a000+22000]
13[15553]: segfault at 0 ip 00000000004043f0 sp 00007ffe805cc098 error 6 in 14[400000+6000]
2[17682]: segfault at 1c ip 00000000004017b0 sp 00007ffdef699940 error 6 in 13 (deleted)[400000+6000]
1[22670]: segfault at 0 ip 0000000000403e5f sp 00007fff41873bc0 error 6 in 14[400000+6000]
18[22937]: segfault at 8 ip 00007ff3b231d7e8 sp 00007fffda09ec50 error 4 in ld-2.17.so[7ff3b2312000+22000]
17[26950]: segfault at 8 ip 00007f83982c07e8 sp 00007fff83665790 error 4 in ld-2.17.so[7f83982b5000+22000]
5[31416]: segfault at 8 ip 00007f1a6434f7e8 sp 00007ffdcdc70b80 error 4 in ld-2.17.so[7f1a64344000+22000]
9[17148]: segfault at 8 ip 00007effea5487e8 sp 00007ffc057cbbd0 error 4 in ld-2.17.so[7effea53d000+22000]
2[22655]: segfault at 8 ip 00007fe52904f7e8 sp 00007ffe05f6bcd0 error 4 in ld-2.17.so[7fe529044000+22000]
3[24081]: segfault at 0 ip (null) sp 00007ffda2fc2e38 error 14 in 3[400000+6000]
16[25402]: segfault at 8 ip 00007fdbda4ed7e8 sp 00007fffe0e24460 error 4 in ld-2.17.so[7fdbda4e2000+22000]
8[26472]: segfault at 8 ip 00007f953462d7e8 sp 00007ffd8d69ff60 error 4 in ld-2.17.so[7f9534622000+22000]
19[30097]: segfault at 8 ip 00007f84d512a7e8 sp 00007ffc9bc69fc0 error 4 in ld-2.17.so[7f84d511f000+22000]
10[5827]: segfault at 8 ip 00007f0a0db4c7e8 sp 00007ffe0387b0f0 error 4 in ld-2.17.so[7f0a0db41000+22000]
2[9548]: segfault at 8 ip 00007f183ab0a7e8 sp 00007ffeabb46630 error 4 in ld-2.17.so[7f183aaff000+22000]
4[10773]: segfault at 8 ip 00007fe015aa97e8 sp 00007ffd93d04290 error 4 in ld-2.17.so[7fe015a9e000+22000]
2[11745]: segfault at 8 ip 00007ff440ff27e8 sp 00007ffd66fa2d40 error 4 in ld-2.17.so[7ff440fe7000+22000]
12[13484]: segfault at 8 ip 00007f8106bc57e8 sp 00007fffb7c0a260 error 4 in ld-2.17.so[7f8106bba000+22000]
Lustre: 9269:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9269:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 475 previous similar messages
Lustre: 9269:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9269:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 475 previous similar messages
Lustre: 9269:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9269:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 475 previous similar messages
Lustre: 9269:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9269:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 475 previous similar messages
Lustre: 9269:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9269:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 475 previous similar messages
Lustre: 9269:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9269:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 475 previous similar messages
11[17414]: segfault at 8 ip 00007fbd529a97e8 sp 00007ffe29831460 error 4 in ld-2.17.so[7fbd5299e000+22000]
Lustre: 16371:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x3578:0x0] with magic=0xbd60bd0
Lustre: 16371:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 7 previous similar messages
7[31935]: segfault at 8 ip 00007fbafd77e7e8 sp 00007fff1cb39e80 error 4 in ld-2.17.so[7fbafd773000+22000]
4[31946]: segfault at 8 ip 00007f5c29efa7e8 sp 00007fff2d8dad00 error 4 in ld-2.17.so[7f5c29eef000+22000]
LustreError: 31946:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a9deb7e8: inode [0x200000402:0x6dfd:0x0] mdc close failed: rc = -13
LustreError: 31946:0:(file.c:247:ll_close_inode_openhandle()) Skipped 5 previous similar messages
2[5810]: segfault at 8 ip 00007f980de477e8 sp 00007ffd0c0bd1e0 error 4 in ld-2.17.so[7f980de3c000+22000]
0[11670]: segfault at 8 ip 00007f6fabdf87e8 sp 00007ffc1567de00 error 4 in ld-2.17.so[7f6fabded000+22000]
14[18293]: segfault at 8 ip 00007f9b45f357e8 sp 00007fff8d838370 error 4 in ld-2.17.so[7f9b45f2a000+22000]
16[20004]: segfault at 8 ip 00007fdc776a67e8 sp 00007ffc071432c0 error 4 in ld-2.17.so[7fdc7769b000+22000]
14[22847]: segfault at 406000 ip 0000000000406000 sp 00007ffc8a8b6838 error 14 in 14[606000+1000]
6[22948]: segfault at 8 ip 00007f0a02f777e8 sp 00007ffdeedd8900 error 4 in ld-2.17.so[7f0a02f6c000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 318f00067 PUD 318f01067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix i2c_core virtio_blk serio_raw libata floppy
CPU: 6 PID: 18161 Comm: ll_sa_18142 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880329049280 ti: ffff8802d5968000 task.ti: ffff8802d5968000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802d596bd78 EFLAGS: 00010296
RAX: ffff8802b8f9a710 RBX: 0000000000000008 RCX: 0000000100260023
RDX: 0000000000000026 RSI: ffff8802b8f9aa40 RDI: 0000000000000008
RBP: ffff8802d596bd88 R08: ffff8802dcd37a98 R09: 0000000000000000
R10: ffff8802dcd34c38 R11: ffff8802dcd37a98 R12: 0000000000000000
R13: ffff8802e97c24f0 R14: ffff8802b8f9aa40 R15: ffff8802dcd37a98
FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002ee606000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1109111>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa1108360>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

13[18036]: segfault at 8 ip 00007f994c22a7e8 sp 00007ffe752fa4e0 error 4 in ld-2.17.so[7f994c21f000+22000]
cat (18394) used greatest stack depth: 9600 bytes left
Lustre: 15893:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e8b6d540 x1832238453607808/t4294972370(0) o101->e301d82d-b48d-45d9-8a98-dc3e2eb37110@0@lo:404/0 lens 376/816 e 0 to 0 dl 1747358814 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10872:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10872:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10872:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10872:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10872:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10872:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10872:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10872:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10872:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10872:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10872:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10872:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10872:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10872:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10872:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 16365:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16365:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 16365:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 16365:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16365:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 16365:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16365:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 19 previous similar messages
LustreError: 26119:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ef489bf8: inode [0x200000401:0x34f:0x0] mdc close failed: rc = -13
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 10872:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10872:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10872:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10872:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10872:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 10872:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10872:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10872:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10872:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10872:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 10868:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 25 previous similar messages
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 25 previous similar messages
3[5343]: segfault at 0 ip 0000000000403e5f sp 00007ffc8ce4b5b0 error 6 in 3[400000+6000]
0[6748]: segfault at 8 ip 00007f41c71b17e8 sp 00007ffccf19b450 error 4 in ld-2.17.so[7f41c71a6000+22000]
Lustre: 15991:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15991:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 79 previous similar messages
Lustre: 15991:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15991:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 15991:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15991:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 15991:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15991:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 15991:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15991:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 15991:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15991:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 79 previous similar messages
LustreError: 16571:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ecc6b7e8: inode [0x200000402:0xba5:0x0] mdc close failed: rc = -13
19[19292]: segfault at 8 ip 00007fd3372227e8 sp 00007ffdec530d40 error 4 in ld-2.17.so[7fd337217000+22000]
6[26630]: segfault at 1c ip 00000000004017b0 sp 00007ffee2354c90 error 6 in 19 (deleted)[400000+6000]
17[26770]: segfault at 6f ip 0000000000404fb3 sp 00007ffe4bd0e4b8 error 6 in 17[400000+6000]
Lustre: 3001:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xe7e:0x0] with magic=0xbd60bd0
Lustre: 9735:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xee0:0x0] with magic=0xbd60bd0
Lustre: 9735:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
6[29754]: segfault at 8 ip 00007f784d1147e8 sp 00007ffdb4096be0 error 4 in ld-2.17.so[7f784d109000+22000]
2[30971]: segfault at 8 ip 00007f9cacc167e8 sp 00007ffe2ae49ac0 error 4 in ld-2.17.so[7f9cacc0b000+22000]
9[31517]: segfault at 0 ip 0000000000403e5f sp 00007ffd08c783d0 error 6 in 9[400000+6000]
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 558, rollback = 7
Lustre: 10872:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 93 previous similar messages
Lustre: 15394:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15394:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 95 previous similar messages
Lustre: 15394:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15394:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 95 previous similar messages
Lustre: 15394:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15394:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 95 previous similar messages
Lustre: 15394:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15394:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 95 previous similar messages
Lustre: 15394:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15394:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 95 previous similar messages
4[2523]: segfault at 8 ip 00007f48955057e8 sp 00007ffcd7055f70 error 4 in ld-2.17.so[7f48954fa000+22000]
Lustre: mdt00_001: service thread pid 9734 was inactive for 40.055 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 9734, comm: mdt00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_object_find_lock+0x54/0x170 [mdt]
[<0>] mdt_reint_setxattr+0x133/0x10e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 15157, comm: mdt00_007 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 18026, comm: mdt00_010 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_004: service thread pid 13895 was inactive for 40.124 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_002: service thread pid 9735 was inactive for 40.054 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: mdt00_018: service thread pid 4534 was inactive for 40.090 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: mdt00_003: service thread pid 11526 was inactive for 40.136 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
LustreError: 9409:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8800a7185a40/0xb74af72bb2bc33fb lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x129b:0x0].0x0 bits 0x1b/0x0 rrc: 23 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb74af72bb2bc33df expref: 616 pid: 9732 timeout: 324 lvb_type: 0
Lustre: mdt00_001: service thread pid 9734 completed after 100.493s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 3001:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802ece75d28 ns: mdt-lustre-MDT0000_UUID lock: ffff8802dd2a2200/0xb74af72bb2bc4ce4 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x129b:0x0].0x0 bits 0x1b/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200400000020 nid: 0@lo remote: 0xb74af72bb2bc4cc8 expref: 18 pid: 3001 timeout: 0 lvb_type: 0
Lustre: mdt00_013: service thread pid 3001 completed after 99.745s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 15895 completed after 99.613s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8802ecc6b7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8802ecc6b7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8802ecc6b7e8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_010: service thread pid 18026 completed after 100.310s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 13895 completed after 100.190s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 15157 completed after 100.412s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 9735 completed after 99.611s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 4626:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000402:0x129b:0x0] error: rc = -5
LustreError: 4687:0:(llite_lib.c:2023:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: mdt00_012: service thread pid 18961 completed after 99.639s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 4979:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ecc6b7e8: inode [0x200000401:0x129d:0x0] mdc close failed: rc = -108
LustreError: 4944:0:(symlink.c:75:ll_readlink_internal()) lustre: inode [0x200000401:0x12a2:0x0]: rc = -108
LustreError: 4085:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000402:0x129b:0x0] error -108.
Lustre: mdt00_005: service thread pid 15116 completed after 99.443s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 15132 completed after 99.263s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 18526 completed after 98.133s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_018: service thread pid 4534 completed after 98.637s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 4798 completed after 96.294s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 4476 completed after 98.133s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_019: service thread pid 4657 completed after 97.470s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 9732 completed after 99.198s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 11526 completed after 96.530s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8802ecc6b7e8: Connection restored to 192.168.123.77@tcp (at 0@lo)
Lustre: mdt00_008: service thread pid 15893 completed after 98.420s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 16365:0:(osd_internal.h:1332:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 16365:0:(osd_internal.h:1332:osd_trans_exec_op()) Skipped 13 previous similar messages
2[8383]: segfault at 8 ip 00007f16cce0b7e8 sp 00007ffef1bcc810 error 4 in ld-2.17.so[7f16cce00000+22000]
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10868:0:(osd_handler.c:1966:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10868:0:(osd_handler.c:1973:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11526:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1435:0x0] with magic=0xbd60bd0
Lustre: 11526:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 10868:0:(osd_handler.c:1983:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10868:0:(osd_handler.c:1990:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10868:0:(osd_handler.c:1997:osd_trans_dump_creds()) Skipped 15 previous similar messages
4[11439]: segfault at 8 ip 00007f2c1822b7e8 sp 00007ffeda81b9f0 error 4 in ld-2.17.so[7f2c18220000+22000]
traps: 0[12615] general protection ip:404e66 sp:7fffffb52f58 error:0 in 0[400000+6000]
traps: 2[12645] general protection ip:404f28 sp:7fff4bf10a78 error:0 in 2[400000+6000]
7[21250]: segfault at 8 ip 00007f1bd6f9f7e8 sp 00007ffd7701ebb0 error 4 in ld-2.17.so[7f1bd6f94000+22000]
LustreError: 21250:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ef489bf8: inode [0x200000403:0x4a0:0x0] mdc close failed: rc = -13
LustreError: 21250:0:(file.c:247:ll_close_inode_openhandle()) Skipped 9 previous similar messages
hrtimer: interrupt took 47167480 ns
Lustre: 4798:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1839:0x0] with magic=0xbd60bd0
Lustre: 4798:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
3[2529]: segfault at 8 ip 00007fe0fd02a7e8 sp 00007fffe793aa70 error 4 in ld-2.17.so[7fe0fd01f000+22000]
LustreError: 8939:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ef489bf8: inode [0x200000403:0xba8:0x0] mdc close failed: rc = -13
Lustre: 9732:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xc0b:0x0] with magic=0xbd60bd0
Lustre: 9732:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
14[12265]: segfault at 8 ip 00007fa9b47707e8 sp 00007ffef35e71d0 error 4 in ld-2.17.so[7fa9b4765000+22000]
19[14574]: segfault at 0 ip 0000000000403e5f sp 00007ffd1c549920 error 6 in 10[400000+6000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
PGD a5fe1067 PUD 960b2067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel squashfs lrw gf128mul glue_helper ablk_helper cryptd pcspkr i2c_piix4 i2c_core binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix libata
CPU: 3 PID: 13048 Comm: ll_sa_12987 Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
task: ffff88009c23e660 ti: ffff8800a37d4000 task.ti: ffff8800a37d4000
RIP: 0010:[<ffffffff813f40ac>] [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8800a37d7d78 EFLAGS: 00010296
RAX: ffff88009331c8d0 RBX: 0000000000000008 RCX: 00000000000ca149
RDX: 00000000000ca148 RSI: ffff88009331cc00 RDI: 0000000000000008
RBP: ffff8800a37d7d88 R08: ffff8800a37d7fd8 R09: 002dffd3a255376d
R10: 0000000000000006 R11: ffff8800a8762a80 R12: 0000000000000000
R13: ffff88009d679110 R14: ffff88009331cc00 R15: ffff88009376d200
FS: 0000000000000000(0000) GS:ffff88013e380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000009377e000 CR4: 0000000000160fe0
Call Trace:
[<ffffffffa10c3481>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa10c26d0>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817f3e5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

random: crng init done
LustreError: 14982:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b7c16800: inode [0x200000401:0xa2:0x0] mdc close failed: rc = -116
dd (15338) used greatest stack depth: 10592 bytes left
LustreError: 23523:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b7c16800: inode [0x200000402:0x523:0x0] mdc close failed: rc = -13
LustreError: 23523:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
19[29462]: segfault at 0 ip 0000000000403e5f sp 00007ffeed1681b0 error 6 in 19[400000+6000]
LustreError: 31250:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3783800: inode [0x200000401:0x858:0x0] mdc close failed: rc = -13
15[10619]: segfault at 0 ip 0000000000403e5f sp 00007ffc8e6b31e0 error 6 in 15[400000+6000]
3[20551]: segfault at 8 ip 00007f37f67827e8 sp 00007ffcdef94430 error 4 in ld-2.17.so[7f37f6777000+22000]
19[22003]: segfault at 8 ip 00007fd3540d07e8 sp 00007fffe202f950 error 4 in ld-2.17.so[7fd3540c5000+22000]
LustreError: 32049:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b3783800: inode [0x200000401:0x17b9:0x0] mdc close failed: rc = -116
LustreError: 32049:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
19[10809]: segfault at 0 ip 0000000000403e5f sp 00007ffdc27adb10 error 6 in 17[400000+6000]
LustreError: 15884:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b7c16800: inode [0x200000402:0x1cec:0x0] mdc close failed: rc = -116
0[26100]: segfault at 8 ip 00007f9468b957e8 sp 00007fffafb9cbb0 error 4 in ld-2.17.so[7f9468b8a000+22000]
16[26484]: segfault at 8 ip 00007f07e07757e8 sp 00007ffdd6aeea10 error 4 in ld-2.17.so[7f07e076a000+22000]
LustreError: 26484:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b7c16800: inode [0x200000402:0x2230:0x0] mdc close failed: rc = -13
LustreError: 26484:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 8c021067 PUD 8c022067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy
CPU: 6 PID: 12405 Comm: ll_sa_12282 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88029ae5b760 ti: ffff88027e070000 task.ti: ffff88027e070000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88027e073d78 EFLAGS: 00010296
RAX: ffff88026bf73090 RBX: 0000000000000008 RCX: 000000010026001c
RDX: 0000000000000026 RSI: ffff88026bf733c0 RDI: 0000000000000008
RBP: ffff88027e073d88 R08: ffff8802a910d978 R09: 0000000000000000
R10: ffff8802a910b858 R11: ffff8802a910d978 R12: 0000000000000000
R13: ffff880299c049d0 R14: ffff88026bf733c0 R15: ffff8802a910d978
FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000a5408000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa110e431>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa110d680>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 18113:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 509 < left 699, rollback = 2
Lustre: 18113:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 18113:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
cat (18350) used greatest stack depth: 10112 bytes left
Lustre: 18113:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 18113:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 18113:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
getfattr (18513) used greatest stack depth: 9936 bytes left
Lustre: 18113:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 816, rollback = 2
Lustre: 18113:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 18113:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 18113:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18113:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 12/816/0
Lustre: 18113:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18113:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 18113:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18113:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 12/215/3, delete: 1/1/0
Lustre: 18113:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18113:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 18113:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 511 < left 699, rollback = 2
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 11/239/2, delete: 0/0/0
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 18113:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/7 is open, migrate only dentry
LustreError: 11497:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x87:0x0]/11 failed: rc = -16
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/5 is open, migrate only dentry
16[20642]: segfault at 8 ip 00007f4fb3eea7e8 sp 00007ffe01ba1070 error 4 in ld-2.17.so[7f4fb3edf000+22000]
13[21291]: segfault at 0 ip 0000000000403e5f sp 00007ffd2e2234e0 error 6 in 13[400000+6000]
LustreError: 21152:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 21152:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: 11496:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/2 is open, migrate only dentry
Lustre: 11496:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 1 previous similar message
LustreError: 21424:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 21424:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 6 previous similar messages
LustreError: 21424:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 21424:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 6 previous similar messages
1[22156]: segfault at 8 ip 00007f8e1440b7e8 sp 00007ffd286c2360 error 4 in ld-2.17.so[7f8e14400000+22000]
LustreError: 22141:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 22141:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 22141:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 22141:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 4 previous similar messages
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/18 is open, migrate only dentry
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 3 previous similar messages
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 502 < left 877, rollback = 2
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/9, destroy: 0/0/0
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 12/275/4, delete: 0/0/0
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: dir [0x200000403:0x80:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: 17407:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a0759440 x1831405848290560/t4294968508(0) o101->ffc3eb26-de74-4fa5-a5e7-64173a3e2d92@0@lo:639/0 lens 376/840 e 0 to 0 dl 1746564789 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 23084:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 23084:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 6 previous similar messages
LustreError: 23084:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 23084:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 6 previous similar messages
Lustre: 20991:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0001: [0x240000403:0x1:0x0]/1 is open, migrate only dentry
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xa:0x0]: rc = -2
LustreError: 24023:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x240000404:0xa:0x0] mdc close failed: rc = -2
7[23866]: segfault at 8 ip 00007f2acb2667e8 sp 00007ffcf89f8d20 error 4 in ld-2.17.so[7f2acb25b000+22000]
Lustre: 20991:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/18 is open, migrate only dentry
Lustre: 20991:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 2 previous similar messages
Lustre: dir [0x280000403:0x6e:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: 22465:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 498 < left 727, rollback = 2
Lustre: 22465:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 113 previous similar messages
Lustre: 22465:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/12, destroy: 0/0/0
Lustre: 22465:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 113 previous similar messages
Lustre: 22465:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 11/727/0
Lustre: 22465:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 113 previous similar messages
Lustre: 22465:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/1
Lustre: 22465:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 113 previous similar messages
Lustre: 22465:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 14/287/4, delete: 1/1/0
Lustre: 22465:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 113 previous similar messages
Lustre: 22465:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 1/1/0
Lustre: 22465:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 113 previous similar messages
LustreError: 26055:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 26055:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 8 previous similar messages
LustreError: 26055:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 26055:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 8 previous similar messages
Lustre: 20340:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802ce662d40 x1831405850289408/t4294969639(0) o101->ffc3eb26-de74-4fa5-a5e7-64173a3e2d92@0@lo:645/0 lens 376/864 e 0 to 0 dl 1746564795 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 18581:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 15' to finish migration: rc = -1
LustreError: 18581:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/15 failed: rc = -1
Lustre: dir [0x240000403:0xb8:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
Lustre: 13518:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 13517:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 13517:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
6[26841]: segfault at 8 ip 00007f2bf17097e8 sp 00007ffd61b70840 error 4 in ld-2.17.so[7f2bf16fe000+22000]
Lustre: 17806:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 17806:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 2 previous similar messages
LustreError: 20991:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000404:0xdc:0x0]/16 failed: rc = -116
6[28401]: segfault at 8 ip 00007f3e25a237e8 sp 00007ffeaad833c0 error 4 in ld-2.17.so[7f3e25a18000+22000]
LustreError: 28401:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x240000404:0x151:0x0] mdc close failed: rc = -13
LustreError: 11487:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x6e:0x0]: rc = -2
Lustre: 11496:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/12 is open, migrate only dentry
Lustre: 11496:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 7 previous similar messages
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 509 < left 1784, rollback = 2
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 196 previous similar messages
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/3, destroy: 1/4/0
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 206 previous similar messages
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 21/1784/0
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 206 previous similar messages
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/129/0
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 206 previous similar messages
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 14/287/3, delete: 2/5/0
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 206 previous similar messages
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 2/2/0
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 206 previous similar messages
Lustre: 17301:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0xf9:0x0] with magic=0xbd60bd0
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 4 previous similar messages
LustreError: 29147:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x40:0x0]: rc = -5
LustreError: 29147:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 47 previous similar messages
LustreError: 29332:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 29332:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 50 previous similar messages
LustreError: 11496:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x153:0x0]/14 failed: rc = -2
LustreError: 11496:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 20991:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '0' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 0' to finish migration: rc = -1
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x115:0x0]: rc = -2
LustreError: 29383:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x280000404:0x115:0x0] mdc close failed: rc = -2
LustreError: 29383:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 28020:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '10' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 10' to finish migration: rc = -1
LustreError: 17405:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
0[6907]: segfault at 8 ip 00007f0d34b057e8 sp 00007ffdc1633610 error 4 in ld-2.17.so[7f0d34afa000+22000]
12[11952]: segfault at 0 ip (null) sp 00007ffc913883f8 error 14 in 12[400000+6000]
LustreError: 10927:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a70f4138: inode [0x280000403:0x211:0x0] mdc close failed: rc = -13
Lustre: 22465:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 510 < left 2140, rollback = 2
Lustre: 22465:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 131 previous similar messages
Lustre: 22465:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/3, destroy: 2/8/0
Lustre: 22465:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 139 previous similar messages
Lustre: 22465:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 25/2140/0
Lustre: 22465:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 139 previous similar messages
Lustre: 22465:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/129/0
Lustre: 22465:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 139 previous similar messages
Lustre: 22465:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 15/323/1, delete: 9/15/1
Lustre: 22465:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 139 previous similar messages
Lustre: 22465:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 9/9/0
Lustre: 22465:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 139 previous similar messages
Lustre: 22465:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/1 is open, migrate only dentry
Lustre: 22465:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 3 previous similar messages
LustreError: 12056:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x117:0x0]: rc = -5
LustreError: 12056:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 34 previous similar messages
LustreError: 12056:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 12056:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 31 previous similar messages
19[13531]: segfault at 8 ip 00007eff5dca17e8 sp 00007ffda643b5b0 error 4 in ld-2.17.so[7eff5dc96000+22000]
LustreError: 29262:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x2ae:0x0]: rc = -2
LustreError: 14676:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a70f4138: inode [0x200000403:0x2ae:0x0] mdc close failed: rc = -2
14[14585]: segfault at 8 ip 00007f89ea8f57e8 sp 00007fff38244170 error 4 in ld-2.17.so[7f89ea8ea000+22000]
LustreError: 29262:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x1d1:0x0]: rc = -2
4[16913]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffef12f59f0 error 14
LustreError: 26491:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000404:0x3ad:0x0]/18 failed: rc = -16
LustreError: 26491:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 3 previous similar messages
LustreError: 15253:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x240000404:0x167:0x0] mdc close failed: rc = -2
LustreError: 15253:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 17806:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 618, rollback = 7
Lustre: 17806:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 7 previous similar messages
LustreError: 22465:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '19' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 19' to finish migration: rc = -1
LustreError: 22465:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x1:0x0]/19 failed: rc = -1
LustreError: 22465:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 4 previous similar messages
5[20610]: segfault at 8 ip 00007efeb400d7e8 sp 00007ffd9c8d1aa0 error 4 in ld-2.17.so[7efeb4002000+22000]
Lustre: 25367:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 618, rollback = 7
Lustre: 25367:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 5 previous similar messages
LustreError: 13888:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x3ad:0x0] migrate mdt count mismatch 1 != 2
14[21757]: segfault at 0 ip 0000000000403cf0 sp 00007ffd7d113678 error 6 in 14[400000+6000]
14[22970]: segfault at 8 ip 00007fd63c0607e8 sp 00007ffcdefc7d80 error 4 in ld-2.17.so[7fd63c055000+22000]
LustreError: 27342:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '19' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 19' to finish migration: rc = -1
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x200000403:0x1:0x0]/2 is open, migrate only dentry
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 15 previous similar messages
Lustre: dir [0x240000404:0x46b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 26491:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 877, rollback = 2
Lustre: 26491:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 569 previous similar messages
Lustre: 26491:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0
Lustre: 26491:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 583 previous similar messages
Lustre: 26491:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 26491:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 583 previous similar messages
Lustre: 26491:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 26491:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 583 previous similar messages
Lustre: 26491:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0
Lustre: 26491:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 583 previous similar messages
Lustre: 26491:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 26491:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 583 previous similar messages
LustreError: 21742:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x64b:0x0]: rc = -2
LustreError: 21742:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 24598:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a70f4138: inode [0x200000404:0x64b:0x0] mdc close failed: rc = -2
LustreError: 16657:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x66d:0x0]: rc = -5
LustreError: 16657:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 16 previous similar messages
LustreError: 16657:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 16657:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 16 previous similar messages
LustreError: 21034:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a70f4138: cannot apply new layout on [0x200000404:0x66d:0x0] : rc = -5
LustreError: 21034:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000404:0x66d:0x0] error -5.
Lustre: dir [0x200000404:0x68e:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 9 previous similar messages
LustreError: 26020:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000404:0x177:0x0]/3 failed: rc = -2
LustreError: 26020:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 3 previous similar messages
2[28406]: segfault at 8 ip 00007fb9b55427e8 sp 00007ffeef7cfce0 error 4 in ld-2.17.so[7fb9b5537000+22000]
3[29113]: segfault at 8 ip 00007ff489c0e7e8 sp 00007ffc6476b930 error 4 in ld-2.17.so[7ff489c03000+22000]
5[28807]: segfault at 8 ip 00007f2b60e197e8 sp 00007ffc60149a00 error 4 in ld-2.17.so[7f2b60e0e000+22000]
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x177:0x0]: rc = -2
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) Skipped 2 previous similar messages
9[30175]: segfault at 8 ip 00007f090311b2fc sp 00007ffdabfe0a30 error 4 in ld-2.17.so[7f090310e000+22000]
Lustre: dir [0x280000403:0x4e7:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 26491:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0002: '10' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 10' to finish migration: rc = -1
LustreError: 26491:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 11498:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 746:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x240000403:0x63f:0x0] : rc = -5
LustreError: 746:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000403:0x63f:0x0] error -5.
LustreError: 1866:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a70f4138: inode [0x240000404:0x70a:0x0] mdc close failed: rc = -2
LustreError: 1866:0:(file.c:247:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: 21775:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 21775:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 7 previous similar messages
8[4895]: segfault at 8 ip 00007fd796de17e8 sp 00007ffdc504d410 error 4 in ld-2.17.so[7fd796dd6000+22000]
LustreError: 27342:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/1 failed: rc = -1
LustreError: 27342:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 10 previous similar messages
8[5083]: segfault at 8 ip 00007fb5094e77e8 sp 00007ffd99f30800 error 4 in ld-2.17.so[7fb5094dc000+22000]
LustreError: 27432:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '17' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 17' to finish migration: rc = -1
LustreError: 27432:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 4 previous similar messages
Lustre: 20350:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xa8d:0x0] with magic=0xbd60bd0
Lustre: 20350:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: 17265:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802bd6a6940 x1831405902623872/t4294994795(0) o101->ffc3eb26-de74-4fa5-a5e7-64173a3e2d92@0@lo:57/0 lens 376/816 e 0 to 0 dl 1746564962 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 28020:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000404:0x6d7:0x0]/3 is open, migrate only dentry
Lustre: 28020:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 26 previous similar messages
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 788, rollback = 2
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1233 previous similar messages
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1265 previous similar messages
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1265 previous similar messages
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/30/0
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1265 previous similar messages
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 12/275/3, delete: 0/0/0
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1265 previous similar messages
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1265 previous similar messages
Lustre: mdt00_035: service thread pid 12836 was inactive for 40.069 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 12836, comm: mdt00_035 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xabc/0xfd0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xb00 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x3c4/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_link+0x8d4/0xf40 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 17291, comm: mdt00_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_intent_getxattr+0x80/0x330 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 12786, comm: mdt00_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_017: service thread pid 17405 was inactive for 40.142 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_028: service thread pid 24463 was inactive for 40.144 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_007: service thread pid 17237 was inactive for 40.086 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: mdt_io00_012: service thread pid 28020 was inactive for 40.123 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: mdt00_012: service thread pid 17301 was inactive for 72.040 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802a4953100/0xf9603e4943377c56 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x70c:0x0].0x0 bits 0x1b/0x0 rrc: 14 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e4943377c10 expref: 434 pid: 24461 timeout: 318 lvb_type: 0
Lustre: mdt00_032: service thread pid 11997 completed after 99.592s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 17237 completed after 99.494s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_035: service thread pid 12836 completed after 100.375s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 17291 completed after 100.324s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_028: service thread pid 24463 completed after 100.067s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0002-mdc-ffff8800a731a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a731a548: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8800a731a548: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
Lustre: mdt00_003: service thread pid 12786 completed after 100.135s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 17405 completed after 100.084s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_031: service thread pid 11996 completed after 99.648s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 11485 completed after 100.040s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 17391 completed after 98.994s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 16978 completed after 99.399s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_038: service thread pid 20854 completed after 99.887s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 13845:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x280000404:0xdc:0x0] mdc close failed: rc = -108
LustreError: 8219:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x70c:0x0] error: rc = -5
Lustre: lustre-MDT0002-mdc-ffff8800a731a548: Connection restored to 192.168.123.82@tcp (at 0@lo)
Lustre: dir [0x280000404:0x502:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 8 previous similar messages
Lustre: mdt_io00_012: service thread pid 28020 completed after 88.139s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_036: service thread pid 13888 completed after 87.823s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 17265 completed after 87.739s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_033: service thread pid 12018 completed after 87.777s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_030: service thread pid 5363 completed after 87.832s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_034: service thread pid 12029 completed after 87.819s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_012: service thread pid 17301 completed after 87.822s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 20321 completed after 87.748s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 17318 completed after 87.774s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 10320:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x9f1:0x0]: rc = -5
LustreError: 10320:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 21 previous similar messages
LustreError: 10320:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 10320:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 21 previous similar messages
3[13921]: segfault at 8 ip 00007fa1d65fd7e8 sp 00007ffc77523a60 error 4 in ld-2.17.so[7fa1d65f2000+22000]
LustreError: 26020:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x280000404:0x798:0x0]/12 failed: rc = -2
LustreError: 26020:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 10 previous similar messages
LustreError: 17922:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x6d6:0x0]: rc = -2
LustreError: 17922:0:(mdd_object.c:3901:mdd_close()) Skipped 3 previous similar messages
Lustre: 23563:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x8e8:0x0] with magic=0xbd60bd0
Lustre: 23563:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 13833:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 13833:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 29 previous similar messages
Lustre: 24461:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xaf3:0x0] with magic=0xbd60bd0
Lustre: 24461:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
traps: 8[17805] trap invalid opcode ip:404c26 sp:7ffeaffad6f8 error:0 in 8[400000+6000]
LustreError: 18581:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '15' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 15' to finish migration: rc = -1
LustreError: 18581:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
Lustre: 20858:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0xb48:0x0] with magic=0xbd60bd0
Lustre: 20858:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 17391:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
3[24909]: segfault at 406000 ip 0000000000406000 sp 00007ffd9b5ad0d8 error 14 in 3[606000+1000]
Lustre: 5649:0:(out_handler.c:879:out_tx_end()) lustre-MDT0002-osd: error during execution of #1 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:433: rc = -61
LustreError: 5649:0:(out_handler.c:889:out_tx_end()) lustre-MDT0002-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:433: rc = -524
1[25307]: segfault at 8 ip 00007f6fe68777e8 sp 00007ffcc7e59cf0 error 4 in ld-2.17.so[7f6fe686c000+22000]
Lustre: 24505:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xd00:0x0] with magic=0xbd60bd0
Lustre: 24505:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: dir [0x200000403:0xd88:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 8 previous similar messages
18[31771]: segfault at 0 ip 0000000000403cf0 sp 00007ffe6d6da738 error 6 in 18[400000+6000]
16[32075]: segfault at 8 ip 00007fb3756b87e8 sp 00007fff2d8bb6e0 error 4 in ld-2.17.so[7fb3756ad000+22000]
Lustre: 2687:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 504 < left 1990, rollback = 2
Lustre: 2687:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 489 previous similar messages
Lustre: 2687:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/7, destroy: 1/4/0
Lustre: 2687:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 517 previous similar messages
Lustre: 2687:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 4/4/0, xattr_set: 25/1990/0
Lustre: 2687:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 517 previous similar messages
Lustre: 2687:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/129/0
Lustre: 2687:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 517 previous similar messages
Lustre: 2687:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 15/263/4, delete: 3/6/0
Lustre: 2687:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 517 previous similar messages
Lustre: 2687:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 8/8/0, ref_del: 3/3/0
Lustre: 2687:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 517 previous similar messages
Lustre: 31575:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/3 is open, migrate only dentry
Lustre: 31575:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 15 previous similar messages
9[3129]: segfault at 8 ip 00007f103ff1b7e8 sp 00007ffc7b77e730 error 4 in ld-2.17.so[7f103ff10000+22000]
LustreError: 211:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 14 [0x200000403:0xe88:0x0] inode@0000000000000000: rc = -5
Lustre: 17405:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0x30a:0x0] with magic=0xbd60bd0
Lustre: 17405:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 5 previous similar messages
7[14829]: segfault at 8 ip 00007f8bd745f7e8 sp 00007ffdf2a8ef00 error 4 in ld-2.17.so[7f8bd7454000+22000]
2[15948]: segfault at 8 ip 00007fb2bd7f47e8 sp 00007ffd5c7edae0 error 4 in ld-2.17.so[7fb2bd7e9000+22000]
1[15973]: segfault at 8 ip 00007efcaa27c7e8 sp 00007ffec79e2990 error 4 in ld-2.17.so[7efcaa271000+22000]
14[17294]: segfault at 8 ip 00007fe6cdd902fc sp 00007ffc6dd24fe0 error 4 in ld-2.17.so[7fe6cdd83000+22000]
6[17636]: segfault at 406000 ip 0000000000406000 sp 00007ffccdc8c478 error 14 in 6[606000+1000]
0[18540]: segfault at 8 ip 00007f8f81d8f7e8 sp 00007ffcf9f25b60 error 4 in ld-2.17.so[7f8f81d84000+22000]
LustreError: 10567:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X0.
11[20757]: segfault at 8 ip 00007ffae4fa37e8 sp 00007ffe7a29f270 error 4 in ld-2.17.so[7ffae4f98000+22000]
Lustre: 28020:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x1:0x0]/3 is open, migrate only dentry
8[22810]: segfault at 8 ip 00007ff6459f87e8 sp 00007fff364e6910 error 4 in ld-2.17.so[7ff6459ed000+22000]
17[23465]: segfault at 404594 ip 0000000000404c98 sp 00007ffd169bb7c8 error 7 in 17[400000+6000]
LustreError: 27432:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '10' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 10' to finish migration: rc = -1
LustreError: 27432:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
19[24977]: segfault at 8 ip 00007fe25b72e7e8 sp 00007ffc2e73a3d0 error 4 in ld-2.17.so[7fe25b723000+22000]
12[25298]: segfault at 8 ip 00007f4d5f94b7e8 sp 00007ffd1696f7d0 error 4 in ld-2.17.so[7f4d5f940000+22000]
14[25008]: segfault at 8 ip 00007f713c29d7e8 sp 00007ffe18fe9be0 error 4 in ld-2.17.so[7f713c292000+22000]
LustreError: 24615:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x200000403:0x1039:0x0] : rc = -5
LustreError: 24615:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1039:0x0] error -5.
Lustre: 15364:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0xb6f:0x0] with magic=0xbd60bd0
Lustre: 15364:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 39 previous similar messages
LustreError: 11487:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0xba9:0x0]: rc = -2
LustreError: 11487:0:(mdd_object.c:3901:mdd_close()) Skipped 5 previous similar messages
7[26183]: segfault at 8 ip 00007f52aa85a7e8 sp 00007fff3c8879f0 error 4 in ld-2.17.so[7f52aa84f000+22000]
LustreError: 28422:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a70f4138: cannot apply new layout on [0x280000405:0x63a:0x0] : rc = -5
LustreError: 28422:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000405:0x63a:0x0] error -5.
LustreError: 20264:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x12c9:0x0] migrate mdt count mismatch 3 != 2
18[32034]: segfault at 8 ip 00007f0ee8bdf7e8 sp 00007fff15901de0 error 4 in ld-2.17.so[7f0ee8bd4000+22000]
18[31857]: segfault at 8 ip 00007f9dc96bb7e8 sp 00007ffc3b7d0e40 error 4 in ld-2.17.so[7f9dc96b0000+22000]
8[32766]: segfault at 8 ip 00007eff8f7f07e8 sp 00007ffdac8d1ec0 error 4 in ld-2.17.so[7eff8f7e5000+22000]
LustreError: 20264:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0xe25:0x0] migrate mdt count mismatch 2 != 1
14[4324]: segfault at 8 ip 00007fb1e05a47e8 sp 00007ffcd3eae530 error 4 in ld-2.17.so[7fb1e0599000+22000]
LustreError: 17318:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0xdde:0x0] migrate mdt count mismatch 1 != 3
14[3127]: segfault at 8 ip 00007f7cbd7df7e8 sp 00007ffc7e9aca30 error 4 in ld-2.17.so[7f7cbd7d4000+22000]
Lustre: dir [0x280000403:0xe2c:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 17 previous similar messages
LustreError: 6793:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xf70:0x0]: rc = -5
LustreError: 6793:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 77 previous similar messages
LustreError: 6793:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 6793:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 77 previous similar messages
LustreError: 3163:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x200000404:0x12be:0x0] mdc close failed: rc = -2
LustreError: 3163:0:(file.c:247:ll_close_inode_openhandle()) Skipped 37 previous similar messages
6[8635]: segfault at 8 ip 00007fa65775d7e8 sp 00007ffc49b617c0 error 4 in ld-2.17.so[7fa657752000+22000]
Lustre: 21775:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 528, rollback = 7
Lustre: 21775:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 133 previous similar messages
6[8909]: segfault at 8 ip 00007f82ae9587e8 sp 00007ffc353ea330 error 4 in ld-2.17.so[7f82ae94d000+22000]
2[9761]: segfault at 8 ip 00007f8cfbf9d7e8 sp 00007fff0dd13280 error 4 in ld-2.17.so[7f8cfbf92000+22000]
LustreError: 27432:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000403:0x1:0x0]/1 failed: rc = -1
LustreError: 27432:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 30 previous similar messages
16[10702]: segfault at 1c ip 00000000004017b0 sp 00007ffd698503f0 error 6 in 16[400000+6000]
8[12521]: segfault at 8 ip 00007f49184f77e8 sp 00007ffee4bd22a0 error 4 in ld-2.17.so[7f49184ec000+22000]
LustreError: 20343:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
LustreError: 171:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 11 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 171:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
5[20013]: segfault at 8 ip 00007f80eb8117e8 sp 00007ffe680edef0 error 4 in ld-2.17.so[7f80eb806000+22000]
LustreError: 20350:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
Lustre: 17301:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0xae3:0x0] with magic=0xbd60bd0
Lustre: 17301:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 19 previous similar messages
17[21773]: segfault at 8 ip 00007f0a2d8117e8 sp 00007ffe1c70e5c0 error 4 in ld-2.17.so[7f0a2d806000+22000]
LustreError: 127:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 14 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 127:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 2 previous similar messages
LustreError: 22100:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a70f4138: cannot apply new layout on [0x240000404:0x1156:0x0] : rc = -5
LustreError: 22100:0:(lov_object.c:1348:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 22100:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000404:0x1156:0x0] error -5.
LustreError: 22100:0:(vvp_io.c:1905:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 11483:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x102f:0x0] migrate mdt count mismatch 1 != 3
0[27521]: segfault at 8 ip 00007f92dc9037e8 sp 00007fff4b1f23c0 error 4 in ld-2.17.so[7f92dc8f8000+22000]
LustreError: 435:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 20 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
0[30130]: segfault at 8 ip 00007f8bdd7487e8 sp 00007ffcb3f56b70 error 4 in ld-2.17.so[7f8bdd73d000+22000]
0[31860]: segfault at 8 ip 00007f76c6ecf7e8 sp 00007fff594b8b70 error 4 in ld-2.17.so[7f76c6ec4000+22000]
LustreError: 1637:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x200000403:0x17a5:0x0] : rc = -5
LustreError: 1637:0:(lov_object.c:1348:lov_layout_change()) Skipped 1 previous similar message
LustreError: 26533:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x200000404:0x1470:0x0] inode@0000000000000000: rc = -5
LustreError: 26533:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 2245:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x240000404:0x1064:0x0] get parent: rc = -116
traps: 15[6596] trap invalid opcode ip:404bae sp:7ffed7fe1508 error:0 in 15[400000+6000]
17[7085]: segfault at 8 ip 00007f44c951b7e8 sp 00007ffce880bae0 error 4 in ld-2.17.so[7f44c9510000+22000]
LustreError: 434:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 7 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 434:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 4 previous similar messages
16[13813]: segfault at 1c ip 00000000004017b0 sp 00007ffc7352e580 error 6 in 11[400000+6000]
LustreError: 15142:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x200000403:0x17a5:0x0] : rc = -5
LustreError: 15142:0:(lov_object.c:1348:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 11497:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '19' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 19' to finish migration: rc = -1
LustreError: 11497:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 20 previous similar messages
LustreError: 13047:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x240000404:0x16e2:0x0] error -5.
LustreError: 13047:0:(vvp_io.c:1905:vvp_io_init()) Skipped 1 previous similar message
LustreError: 20849:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
18[18313]: segfault at 8 ip 00007f60182d97e8 sp 00007ffc98be56f0 error 4 in ld-2.17.so[7f60182ce000+22000]
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88009e1b25c0/0xf9603e49438f46fc lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x1df6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e49438f46e0 expref: 1149 pid: 17291 timeout: 643 lvb_type: 0
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/15 is open, migrate only dentry
Lustre: 11497:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 95 previous similar messages
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 16943:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000404:0x1df6:0x0] error: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection restored to 192.168.123.82@tcp (at 0@lo)
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 506 < left 699, rollback = 2
Lustre: 18581:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3571 previous similar messages
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0
Lustre: 18581:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3771 previous similar messages
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 18581:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3771 previous similar messages
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 18581:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3771 previous similar messages
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 12/275/4, delete: 0/0/0
Lustre: 18581:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3771 previous similar messages
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 18581:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3771 previous similar messages
LustreError: 15799:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0xeab:0x0]: rc = -2
LustreError: 15799:0:(mdd_object.c:3901:mdd_close()) Skipped 13 previous similar messages
2[18461]: segfault at 8 ip 00007f40a54e37e8 sp 00007fff2704af10 error 4 in ld-2.17.so[7f40a54d8000+22000]
Lustre: 28657:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x1490:0x0] with magic=0xbd60bd0
Lustre: 28657:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 39 previous similar messages
15[21362]: segfault at 406000 ip 0000000000406000 sp 00007ffd2a3a29a8 error 14 in 15[606000+1000]
LustreError: 20656:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a70f4138: cannot apply new layout on [0x240000404:0x16e2:0x0] : rc = -5
LustreError: 20656:0:(lov_object.c:1348:lov_layout_change()) Skipped 4 previous similar messages
LustreError: 23888:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000405:0x117a:0x0] error -5.
6[24637]: segfault at 8 ip 00007fee013db7e8 sp 00007ffd70216bb0 error 4 in ld-2.17.so[7fee013d0000+22000]
LustreError: 20854:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1720:0x0] migrate mdt count mismatch 2 != 3
LustreError: 20854:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
LustreError: 11484:0:(mdd_object.c:391:mdd_xattr_get()) lustre-MDD0002: object [0x280000403:0x14f3:0x0] not found: rc = -2
7[30499]: segfault at 8 ip 00007f68bfa2e7e8 sp 00007ffeb3def490 error 4 in ld-2.17.so[7f68bfa23000+22000]
8[31072]: segfault at 8 ip 00007fa0f2ea87e8 sp 00007ffd0adf6910 error 4 in ld-2.17.so[7fa0f2e9d000+22000]
8[31353]: segfault at 8 ip 00007faf5181f7e8 sp 00007ffd090f7890 error 4 in ld-2.17.so[7faf51814000+22000]
Lustre: 13833:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 594, rollback = 7
Lustre: 13833:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 129 previous similar messages
Lustre: lustre-OST0002-osc-ffff8800a731a548: disconnect after 21s idle
Lustre: lustre-OST0003-osc-ffff8800a70f4138: disconnect after 23s idle
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88029a4de940/0xf9603e49439bd067 lrc: 3/0,0 mode: PR/PR res: [0x200000405:0x22f:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e49439bd02f expref: 238 pid: 11483 timeout: 775 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 28402:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x280000403:0x15a9:0x0] : rc = -5
LustreError: 28402:0:(lov_object.c:1348:lov_layout_change()) Skipped 7 previous similar messages
LustreError: 28402:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000403:0x15a9:0x0] error -5.
LustreError: 28402:0:(vvp_io.c:1905:vvp_io_init()) Skipped 2 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 29585:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x200000405:0x2fe:0x0] get parent: rc = -108
LustreError: 30798:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x200000405:0x219:0x0] mdc close failed: rc = -108
LustreError: 30798:0:(file.c:247:ll_close_inode_openhandle()) Skipped 57 previous similar messages
LustreError: 30664:0:(llite_lib.c:1996:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: Skipped 1 previous similar message
LustreError: 31128:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000405:0x2fe:0x0] error: rc = -107
LustreError: 31128:0:(file.c:6143:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection restored to 192.168.123.82@tcp (at 0@lo)
LustreError: 31170:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000405:0x12c2:0x0]: rc = -5
LustreError: 31170:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 203 previous similar messages
LustreError: 31170:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 31170:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 203 previous similar messages
LustreError: 2004:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
Lustre: dir [0x240000404:0x1bf6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 14 previous similar messages
LustreError: 20343:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1827:0x0] migrate mdt count mismatch 1 != 2
10[4710]: segfault at 8 ip 00007f5f6f7e17e8 sp 00007ffe87ae8060 error 4 in ld-2.17.so[7f5f6f7d6000+22000]
17[4490]: segfault at 8 ip 00007ff9452f37e8 sp 00007fffa554b140 error 4 in ld-2.17.so[7ff9452e8000+22000]
9[6711]: segfault at 8 ip 00007f2abb2157e8 sp 00007ffdf8ee5900 error 4 in ld-2.17.so[7f2abb20a000+22000]
11[6919]: segfault at 1c ip 00000000004017b0 sp 00007ffc9d5921a0 error 6 in 11[400000+6000]
17[7561]: segfault at 8 ip 00007fbdbf0577e8 sp 00007ffe8a435b70 error 4 in ld-2.17.so[7fbdbf04c000+22000]
17[7582]: segfault at 8 ip 00007f8165f957e8 sp 00007ffd6a732880 error 4 in ld-2.17.so[7f8165f8a000+22000]
16[11648]: segfault at 8 ip 00007f2fad6a67e8 sp 00007ffe09867750 error 4 in ld-2.17.so[7f2fad69b000+22000]
6[17566]: segfault at 8 ip 00007f671cdca7e8 sp 00007fff99b642d0 error 4 in ld-2.17.so[7f671cdbf000+22000]
7[18921]: segfault at 8 ip 00007f1b4c8e87e8 sp 00007fff5fbf2940 error 4 in ld-2.17.so[7f1b4c8dd000+22000]
4[18819]: segfault at 8 ip 00007f637bc737e8 sp 00007ffeb8843940 error 4 in ld-2.17.so[7f637bc68000+22000]
LustreError: 27432:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/19 failed: rc = -2
LustreError: 27432:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 49 previous similar messages
2[20436]: segfault at 8 ip 00007f4ded15f7e8 sp 00007ffddaaa47a0 error 4 in ld-2.17.so[7f4ded154000+22000]
Lustre: 27342:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x2:0x0]/3 is open, migrate only dentry
Lustre: 27342:0:(mdt_reint.c:2460:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 17302:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1c5e:0x0] migrate mdt count mismatch 2 != 1
18[25462]: segfault at 8 ip 00007f8a2ac8a7e8 sp 00007ffe540d4300 error 4 in ld-2.17.so[7f8a2ac7f000+22000]
LustreError: 20991:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 0' to finish migration: rc = -1
LustreError: 20991:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 7 previous similar messages
19[27277]: segfault at 8 ip 00007efcb74be7e8 sp 00007ffe911512b0 error 4 in ld-2.17.so[7efcb74b3000+22000]
3[30212]: segfault at 8 ip 00007f42b75887e8 sp 00007ffef9e19410 error 4
3[30261]: segfault at 8 ip 00007f15b180f7e8 sp 00007ffeadac41f0 error 4
in ld-2.17.so[7f15b1804000+22000]
in ld-2.17.so[7f42b757d000+22000]
7[30267]: segfault at 8 ip 00007fbe6a4e27e8 sp 00007fffc0d91e80 error 4 in ld-2.17.so[7fbe6a4d7000+22000]
15[32594]: segfault at 8 ip 00007f7862ab57e8 sp 00007ffd457b0da0 error 4 in ld-2.17.so[7f7862aaa000+22000]
19[3839]: segfault at 8 ip 00007f21879037e8 sp 00007ffee14ccd80 error 4 in ld-2.17.so[7f21878f8000+22000]
LustreError: 12029:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
8[5044]: segfault at 8 ip 00007f6a7358b7e8 sp 00007ffd1104c270 error 4 in ld-2.17.so[7f6a73580000+22000]
LustreError: 3453:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x27be:0x0] error -5.
17[7305]: segfault at 8 ip 00007fe05979b7e8 sp 00007ffda28dc450 error 4 in ld-2.17.so[7fe059790000+22000]
traps: 17[7952] trap invalid opcode ip:4059b4 sp:7ffde90be0a8 error:0 in 17[400000+6000]
12[9052]: segfault at 0 ip 0000000000403e5f sp 00007ffc0cf49620 error 6 in 10[400000+6000]
19[7707]: segfault at 0 ip 0000000000403cf0 sp 00007ffc5b4b1118 error 6 in 19[400000+6000]
7[10114]: segfault at 1c ip 0000000000401360 sp 00007ffce2b959e8 error 6 in 14[400000+6000]
Lustre: 31575:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0001: [0x240000403:0x1:0x0]/13 is open, migrate only dentry
LustreError: 14474:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a70f4138: cannot apply new layout on [0x200000406:0x761:0x0] : rc = -5
LustreError: 14474:0:(lov_object.c:1348:lov_layout_change()) Skipped 6 previous similar messages
LustreError: 17995:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000405:0x19ba:0x0]: rc = -2
LustreError: 17995:0:(mdd_object.c:3901:mdd_close()) Skipped 19 previous similar messages
LustreError: 17262:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
LustreError: 17262:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 2 previous similar messages
LustreError: 11485:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000406:0x6ca:0x0] migrate mdt count mismatch 1 != 3
LustreError: 11485:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
7[21082]: segfault at 8 ip 00007f73e344d7e8 sp 00007fffccea57d0 error 4 in ld-2.17.so[7f73e3442000+22000]
Lustre: 20340:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x1fb1:0x0] with magic=0xbd60bd0
Lustre: 20340:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 55 previous similar messages
17[26415]: segfault at 8 ip 00007ff18621e7e8 sp 00007fff16c48540 error 4 in ld-2.17.so[7ff186213000+22000]
13[28141]: segfault at 8 ip 00007ffa51a7c7e8 sp 00007ffd653353e0 error 4 in ld-2.17.so[7ffa51a71000+22000]
1[10047]: segfault at 8 ip 00007ff9952e47e8 sp 00007ffcb7551680 error 4 in ld-2.17.so[7ff9952d9000+22000]
3[11149]: segfault at 8 ip 00007f6466a6a7e8 sp 00007ffd0b881380 error 4 in ld-2.17.so[7f6466a5f000+22000]
LustreError: 241:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 17 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
1[15169]: segfault at 0 ip (null) sp 00007fffde7d3598 error 14 in 1[400000+6000]
17[18577]: segfault at 1c ip 00000000004017b0 sp 00007ffdc617f550 error 6 in 17[400000+6000]
LustreError: 20407:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000403:0x3056:0x0] error -5.
LustreError: 20407:0:(vvp_io.c:1905:vvp_io_init()) Skipped 6 previous similar messages
19[26446]: segfault at 8 ip 00007f2a21b687e8 sp 00007ffe7c8c0430 error 4 in ld-2.17.so[7f2a21b5d000+22000]
10[26123]: segfault at 8 ip 00007f1a445497e8 sp 00007fff065a7380 error 4 in ld-2.17.so[7f1a4453e000+22000]
LustreError: 24174:0:(llite_lib.c:1845:ll_update_lsm_md()) lustre: [0x200000403:0x3161:0x0] dir layout mismatch:
LustreError: 24174:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 24174:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0xbf:0x0]
LustreError: 24174:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
LustreError: 24191:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 24191:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=2 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
15[27185]: segfault at 0 ip 0000000000403e5f sp 00007fff731d1600 error 6 in 15[400000+6000]
LustreError: 12786:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
3[433]: segfault at 8 ip 00007f77a8dc07e8 sp 00007fffa3852770 error 4 in ld-2.17.so[7f77a8db5000+22000]
16[594]: segfault at 8 ip 00007f66255b87e8 sp 00007ffc5a1e9d60 error 4 in ld-2.17.so[7f66255ad000+22000]
0[776]: segfault at 7ffd024cf318 ip 0000000000405c3a sp 00007ffd020cacd8 error 6 in 0[400000+6000]
3[848]: segfault at 8 ip 00007f780d6a77e8 sp 00007ffc3836b470 error 4 in ld-2.17.so[7f780d69c000+22000]
2[2082]: segfault at 0 ip 0000000000403e5f sp 00007ffc42850a10 error 6 in 2[400000+6000]
2[3674]: segfault at 0 ip (null) sp 00007ffd92016e08 error 14 in 2[400000+6000]
7[3728]: segfault at 8 ip 00007f3e62c977e8 sp 00007ffdf7427450 error 4 in ld-2.17.so[7f3e62c8c000+22000]
1[5360]: segfault at 8 ip 00007faf48c137e8 sp 00007ffea2f726a0 error 4 in ld-2.17.so[7faf48c08000+22000]
LustreError: 11996:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000406:0x11c1:0x0] migrate mdt count mismatch 3 != 2
LustreError: 11996:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 4 previous similar messages
Lustre: dir [0x200000403:0x3349:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 51 previous similar messages
9[6883]: segfault at 0 ip 0000000000403cf0 sp 00007ffeb054d768 error 6 in 9[400000+6000]
getfattr (7494) used greatest stack depth: 9792 bytes left
10[13402]: segfault at 8 ip 00007fc8e4eef7e8 sp 00007ffdc0dca500 error 4 in ld-2.17.so[7fc8e4ee4000+22000]
2[14524]: segfault at 8 ip 00007f59829ed7e8 sp 00007ffcc3284f80 error 4 in ld-2.17.so[7f59829e2000+22000]
2[17680]: segfault at 8 ip 00007f04ca7e17e8 sp 00007ffc3a2b6e40 error 4 in ld-2.17.so[7f04ca7d6000+22000]
traps: 18[22086] trap invalid opcode ip:4054a2 sp:7ffe3aad4588 error:0 in 18[400000+6000]
1[22872]: segfault at 8 ip 00007f7e5f5697e8 sp 00007ffec9b39c70 error 4 in ld-2.17.so[7f7e5f55e000+22000]
8[23014]: segfault at 406000 ip 0000000000406000 sp 00007ffce302f6b8 error 14 in 8[606000+1000]
1[23585]: segfault at 8 ip 00007f553c2007e8 sp 00007ffd54b8c720 error 4 in ld-2.17.so[7f553c1f5000+22000]
Lustre: 26020:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 505 < left 877, rollback = 2
Lustre: 26020:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 6350 previous similar messages
Lustre: 26020:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0
Lustre: 26020:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 6646 previous similar messages
Lustre: 26020:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 26020:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 6646 previous similar messages
Lustre: 26020:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 26020:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 6646 previous similar messages
Lustre: 26020:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 11/239/4, delete: 0/0/0
Lustre: 26020:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 6646 previous similar messages
Lustre: 26020:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 26020:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 6646 previous similar messages
Lustre: 18113:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x337c:0x0]/15 is open, migrate only dentry
Lustre: 18113:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 165 previous similar messages
ptlrpc_watchdog_fire: 15 callbacks suppressed
Lustre: ll_ost_out00_00: service thread pid 5649 was inactive for 40.025 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 5649, comm: ll_ost_out00_00 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x2a2/0x750 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_getattr_name_lock+0x1844/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] mdt_batch_getattr+0x6c/0x190 [mdt]
[<0>] mdt_batch+0x465/0x195b [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8802a7d4b100/0xf9603e49440884ee lrc: 3/0,0 mode: PR/PR res: [0x200000406:0x16d1:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e49440884cb expref: 1038 pid: 16978 timeout: 1263 lvb_type: 0
LustreError: 17391:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e73bca88 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cd436580/0xf9603e4944095fe6 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x337c:0x0].0x0 bits 0x12/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200400000020 nid: 0@lo remote: 0xf9603e4944095fd8 expref: 817 pid: 17391 timeout: 0 lvb_type: 0
LustreError: lustre-MDT0000-mdc-ffff8800a70f4138: operation mds_getattr_lock to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800a70f4138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 13543:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746565950 with bad export cookie 17969430997586500074
LustreError: lustre-MDT0000-mdc-ffff8800a70f4138: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 27404:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x337c:0x0] error: rc = -5
Lustre: ll_ost_out00_00: service thread pid 5649 completed after 74.357s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 29147:0:(llite_lib.c:1996:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 28766:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8800a70f4138: [0x200000402:0x90:0x0] lock enqueue fails: rc = -108
LustreError: 1843:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff8800a70f4138: namespace resource [0x200000007:0x1:0x0].0x0 (ffff8802a48f4a40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffff8800a70f4138: Connection restored to 192.168.123.82@tcp (at 0@lo)
LustreError: 17396:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
LustreError: 17396:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 1 previous similar message
Lustre: 12664:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12664:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 275 previous similar messages
LustreError: 3075:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x200000407:0x70:0x0] : rc = -5
LustreError: 3075:0:(lov_object.c:1348:lov_layout_change()) Skipped 34 previous similar messages
2[8040]: segfault at 8 ip 00007f88f1e297e8 sp 00007fff6601bb30 error 4 in ld-2.17.so[7f88f1e1e000+22000]
LustreError: 7088:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x280000403:0x2ca8:0x0] mdc close failed: rc = -2
LustreError: 7088:0:(file.c:247:ll_close_inode_openhandle()) Skipped 89 previous similar messages
LustreError: 9802:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x2e52:0x0]: rc = -5
LustreError: 9802:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 493 previous similar messages
LustreError: 9802:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 9802:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 493 previous similar messages
LustreError: 2004:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 11 [0x240000403:0x2cc1:0x0] inode@0000000000000000: rc = -5
LustreError: 2004:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 4 previous similar messages
LustreError: 9201:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x280000403:0x2d25:0x0] error -5.
LustreError: 9201:0:(vvp_io.c:1905:vvp_io_init()) Skipped 6 previous similar messages
1[10567]: segfault at 8 ip 00007f4c574147e8 sp 00007ffd8aad58f0 error 4 in ld-2.17.so[7f4c57409000+22000]
2[10961]: segfault at 8 ip 00007f64b542c7e8 sp 00007ffc0d1b1c50 error 4 in ld-2.17.so[7f64b5421000+22000]
19[13929]: segfault at 8 ip 00007f2141c507e8 sp 00007ffcb4da8910 error 4 in ld-2.17.so[7f2141c45000+22000]
12[23051]: segfault at 8 ip 00007ff7c6cb57e8 sp 00007ffec29a58a0 error 4 in ld-2.17.so[7ff7c6caa000+22000]
LustreError: 20991:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/5 failed: rc = -1
LustreError: 20991:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 148 previous similar messages
12[27137]: segfault at 8 ip 00007f370044e7e8 sp 00007ffc4cf93a20 error 4 in ld-2.17.so[7f3700443000+22000]
4[28675]: segfault at 8 ip 00007f83d096b7e8 sp 00007fff2e28c9e0 error 4 in ld-2.17.so[7f83d0960000+22000]
LustreError: 20264:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x566c:0x0] migrate mdt count mismatch 1 != 3
LustreError: 20264:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 5 previous similar messages
0[31772]: segfault at 8 ip 00007fc7521467e8 sp 00007fff638a8080 error 4 in ld-2.17.so[7fc75213b000+22000]
LustreError: 31470:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) lustre-MDD0000: '18' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 18' to finish migration: rc = -1
LustreError: 31470:0:(mdd_dir.c:4743:mdd_migrate_cmd_check()) Skipped 61 previous similar messages
12[7565]: segfault at 8 ip 00007f68c9c407e8 sp 00007ffcd3ff1ce0 error 4 in ld-2.17.so[7f68c9c35000+22000]
traps: 9[10722] trap invalid opcode ip:404afe sp:7ffde44ddc98 error:0 in 9[400000+6000]
3[11203]: segfault at 8 ip 00007fb365f3c7e8 sp 00007fff9e503aa0 error 4 in ld-2.17.so[7fb365f31000+22000]
17[12103]: segfault at 8 ip 00007fe4816687e8 sp 00007fffb65f1340 error 4 in ld-2.17.so[7fe48165d000+22000]
13[15514]: segfault at 8 ip 00007fba88ccf7e8 sp 00007ffd4f46bd90 error 4 in ld-2.17.so[7fba88cc4000+22000]
16[17568]: segfault at 8 ip 00007f572eddf7e8 sp 00007ffc24d65be0 error 4 in ld-2.17.so[7f572edd4000+22000]
LustreError: 17265:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x26:0x0] doesn't exist!: rc = -14
LustreError: 17265:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 1 previous similar message
LustreError: 18113:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000407:0xaad:0x0]: rc = -2
LustreError: 11486:0:(mdd_object.c:3901:mdd_close()) Skipped 29 previous similar messages
15[24055]: segfault at 8 ip 00007f2ac21d17e8 sp 00007fffcd1ce0c0 error 4 in ld-2.17.so[7f2ac21c6000+22000]
9[27571]: segfault at 8 ip 00007f5272ba27e8 sp 00007ffe374147e0 error 4 in ld-2.17.so[7f5272b97000+22000]
9[28501]: segfault at 8 ip 00007f90d34057e8 sp 00007ffc59a2aea0 error 4 in ld-2.17.so[7f90d33fa000+22000]
4[30782]: segfault at 0 ip 0000000000403e5f sp 00007ffd22e92100 error 6 in 4[400000+6000]
Lustre: 17318:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0x325d:0x0] with magic=0xbd60bd0
Lustre: 17318:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 217 previous similar messages
6[6370]: segfault at 8 ip 00007fcd838027e8 sp 00007ffd7ceaa2d0 error 4 in ld-2.17.so[7fcd837f7000+22000]
LustreError: 18113:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
1[8072]: segfault at 8 ip 00007fb9ae4b47e8 sp 00007ffcc2ad5f20 error 4 in ld-2.17.so[7fb9ae4a9000+22000]
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802a67acb40/0xf9603e49444b46a3 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x5c54:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e49444b466b expref: 1032 pid: 16978 timeout: 1555 lvb_type: 0
LustreError: 11128:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746566242 with bad export cookie 17969430997586500326
Lustre: lustre-MDT0001-mdc-ffff8800a731a548: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 11997:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802cd79bc40 x1831406271190784/t0(0) o104->lustre-MDT0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-MDT0001-mdc-ffff8800a731a548: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 10851:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff8800a731a548: [0x240000401:0x5f:0x0] lock enqueue fails: rc = -108
LustreError: 10851:0:(mdc_request.c:1469:mdc_read_page()) Skipped 5 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff8800a731a548: Connection restored to 192.168.123.82@tcp (at 0@lo)
4[32527]: segfault at 8 ip 00007f1452b3f7e8 sp 00007fff01d9bc00 error 4 in ld-2.17.so[7f1452b34000+22000]
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880091dc4b40/0xf9603e4944566438 lrc: 3/0,0 mode: CR/CR res: [0x200000406:0x28c1:0x0].0x0 bits 0xa/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e494456641c expref: 1079 pid: 17265 timeout: 1596 lvb_type: 0
LustreError: 11484:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d0b58008 ns: mdt-lustre-MDT0000_UUID lock: ffff8800a3a35680/0xf9603e4944582a74 lrc: 3/0,0 mode: PR/PR res: [0x200000406:0x2840:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xf9603e494456a403 expref: 695 pid: 11484 timeout: 0 lvb_type: 0
LustreError: 11484:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 2687:0:(client.c:1373:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802cdfe2340 x1831406278275840/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-MDT0000-mdc-ffff8800a731a548: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 11653:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 11653:0:(file.c:6143:ll_inode_revalidate_fini()) Skipped 221 previous similar messages
LustreError: 9373:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff8800a731a548: [0x200000401:0x76:0x0] lock enqueue fails: rc = -108
Lustre: dir [0x240000404:0x5cea:0x0] stripe 2 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 32 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a731a548: Connection restored to 192.168.123.82@tcp (at 0@lo)
0[25438]: segfault at 8 ip 00007f441f2c17e8 sp 00007ffdc20d5bc0 error 4 in ld-2.17.so[7f441f2b6000+22000]
traps: 3[25881] general protection ip:404910 sp:7fffefee8fe8 error:0 in 3[400000+6000]
5[27352]: segfault at 8 ip 00007f3aa9c9a7e8 sp 00007ffc4e1f4e20 error 4 in ld-2.17.so[7f3aa9c8f000+22000]
Lustre: 30680:0:(out_handler.c:879:out_tx_end()) lustre-MDT0000-osd: error during execution of #2 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:433: rc = -61
LustreError: 30680:0:(out_handler.c:889:out_tx_end()) lustre-MDT0000-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:410: rc = -524
6[32653]: segfault at 8 ip 00007fb9b5c677e8 sp 00007ffe9a0ba970 error 4 in ld-2.17.so[7fb9b5c5c000+22000]
0[32405]: segfault at 8 ip 00007fa2088997e8 sp 00007ffffcd41e20 error 4 in ld-2.17.so[7fa20888e000+22000]
13[1709]: segfault at 8 ip 00007f4d565437e8 sp 00007ffcfde1b000 error 4 in ld-2.17.so[7f4d56538000+22000]
5[3030]: segfault at 8 ip 00007f91c3abc7e8 sp 00007ffc1e6a1d40 error 4 in ld-2.17.so[7f91c3ab1000+22000]
15[3125]: segfault at 8 ip 00007f9b4b17e7e8 sp 00007ffc2788e600 error 4 in ld-2.17.so[7f9b4b173000+22000]
10[3878]: segfault at 8 ip 00007f7eb92b07e8 sp 00007ffc5a8d69a0 error 4 in ld-2.17.so[7f7eb92a5000+22000]
13[5790]: segfault at 8 ip 00007fe28c7967e8 sp 00007ffe513e1ae0 error 4 in ld-2.17.so[7fe28c78b000+22000]
8[5687]: segfault at 8 ip 00007fad5c46c7e8 sp 00007ffd22c82730 error 4 in ld-2.17.so[7fad5c461000+22000]
13[8569]: segfault at 8 ip 00007f9488e177e8 sp 00007ffd43d48e70 error 4 in ld-2.17.so[7f9488e0c000+22000]
3[10190]: segfault at 8 ip 00007fd1ff8137e8 sp 00007ffd05afd5f0 error 4 in ld-2.17.so[7fd1ff808000+22000]
7[11688]: segfault at 8 ip 00007f11bb2987e8 sp 00007fff191a01f0 error 4 in ld-2.17.so[7f11bb28d000+22000]
LustreError: 17405:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x5d35:0x0] doesn't exist!: rc = -14
LustreError: 17405:0:(mdt_open.c:1302:mdt_cross_open()) Skipped 183 previous similar messages
9[17271]: segfault at 406000 ip 0000000000406000 sp 00007fff7d4f6b08 error 14 in 9[606000+1000]
LustreError: 11132:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88009e2b9e40/0xf9603e4944725757 lrc: 3/0,0 mode: PW/PW res: [0x240000403:0x3e85:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf9603e494472572d expref: 824 pid: 11485 timeout: 1806 lvb_type: 0
Lustre: 27342:0:(mdd_dir.c:4822:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/9 is open, migrate only dentry
Lustre: 27342:0:(mdd_dir.c:4822:mdd_migrate_object()) Skipped 129 previous similar messages
LustreError: 17391:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802e5482548 ns: mdt-lustre-MDT0001_UUID lock: ffff8802c2286580/0xf9603e4944727621 lrc: 3/0,0 mode: PR/PR res: [0x240000405:0x311:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xf9603e4944727605 expref: 559 pid: 17391 timeout: 0 lvb_type: 0
LustreError: lustre-MDT0001-mdc-ffff8800a70f4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff8800a70f4138: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff8800a70f4138: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 17391:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: 22522:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x1:0x0] error: rc = -5
LustreError: 22522:0:(file.c:6143:ll_inode_revalidate_fini()) Skipped 84 previous similar messages
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 513 < left 966, rollback = 2
Lustre: 28020:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 5526 previous similar messages
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0
Lustre: 28020:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 5690 previous similar messages
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 12/966/0
Lustre: 28020:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 5690 previous similar messages
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 4/54/0
Lustre: 28020:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 5690 previous similar messages
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 11/239/1, delete: 0/0/0
Lustre: 28020:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 5690 previous similar messages
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0
Lustre: 28020:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 5690 previous similar messages
LustreError: 23313:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff8800a70f4138: [0x240000401:0x6b:0x0] lock enqueue fails: rc = -108
LustreError: 23313:0:(mdc_request.c:1469:mdc_read_page()) Skipped 3 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff8800a70f4138: Connection restored to 192.168.123.82@tcp (at 0@lo)
traps: 0[23965] trap invalid opcode ip:405152 sp:7ffed333d2b8 error:0 in 0[400000+6000]
Lustre: 11489:0:(out_handler.c:879:out_tx_end()) lustre-MDT0000-osd: error during execution of #0 from /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:433: rc = -61
7[29654]: segfault at 8 ip 00007f766260a7e8 sp 00007ffd57678660 error 4 in ld-2.17.so[7f76625ff000+22000]
11[32130]: segfault at 0 ip 0000000000403e5f sp 00007ffe1b693920 error 6 in 4[400000+6000]
LustreError: 19229:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 4 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 19229:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 2 previous similar messages
8[8295]: segfault at 0 ip 0000000000403cf0 sp 00007ffe0d54ae58 error 6 in 8[400000+6000]
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18247:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 153 previous similar messages
11[21201]: segfault at 0 ip 0000000000403e5f sp 00007ffcec621660 error 6 in 11[400000+6000]
3[22342]: segfault at 8 ip 00007f4cb2ba47e8 sp 00007ffc03a94c60 error 4 in ld-2.17.so[7f4cb2b99000+22000]
LustreError: 25396:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a731a548: inode [0x200000408:0xa07:0x0] mdc close failed: rc = -13
LustreError: 25396:0:(file.c:247:ll_close_inode_openhandle()) Skipped 175 previous similar messages
6[28204]: segfault at 8 ip 00007fd49aac97e8 sp 00007ffe1fd82440 error 4 in ld-2.17.so[7fd49aabe000+22000]
LustreError: 26011:0:(statahead.c:2399:start_statahead_thread()) lustre: invalid pattern 0X0.
0[28461]: segfault at 1c ip 00000000004017b0 sp 00007ffceabbce90 error 6 in 0[400000+6000]
9[31421]: segfault at 8 ip 00007f5c4bf837e8 sp 00007ffc173990a0 error 4 in ld-2.17.so[7f5c4bf78000+22000]
LustreError: 20321:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x373a:0x0] migrate mdt count mismatch 2 != 3
LustreError: 20321:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 7 previous similar messages
LustreError: 3597:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000407:0x1cb7:0x0]: rc = -5
LustreError: 3597:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 233 previous similar messages
LustreError: 3597:0:(llite_lib.c:3700:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 3597:0:(llite_lib.c:3700:ll_prep_inode()) Skipped 233 previous similar messages
LustreError: 3430:0:(lov_object.c:1348:lov_layout_change()) lustre-clilov-ffff8800a731a548: cannot apply new layout on [0x200000407:0x1cb7:0x0] : rc = -5
LustreError: 3430:0:(lov_object.c:1348:lov_layout_change()) Skipped 30 previous similar messages
LustreError: 3430:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000407:0x1cb7:0x0] error -5.
LustreError: 3430:0:(vvp_io.c:1905:vvp_io_init()) Skipped 9 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 31f939067 PUD 2ee86d067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata
CPU: 13 PID: 2258 Comm: ll_sa_2214 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800a6493760 ti: ffff8802c3500000 task.ti: ffff8802c3500000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802c3503d78 EFLAGS: 00010296
RAX: ffff8802c1a0ba10 RBX: 0000000000000008 RCX: 0000000100260025
RDX: 0000000000000026 RSI: ffff8802c1a0bd40 RDI: 0000000000000008
RBP: ffff8802c3503d88 R08: ffff8802db826d58 R09: 0000000000000000
R10: ffff8802db823858 R11: ffff8802db826d58 R12: 0000000000000000
R13: ffff8802e9efa4f0 R14: ffff8802c1a0bd40 R15: ffff8802db826d58
FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002eb9c4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa10cd431>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa10cc680>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

14[15625]: segfault at 8 ip 00007f1a676597e8 sp 00007ffed3752100 error 4 in ld-2.17.so[7f1a6764e000+22000]
Lustre: 9530:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802edd64b40 x1831153276370688/t4294968896(0) o101->32e3ad90-6595-406b-93f0-c86f06af6e10@0@lo:628/0 lens 376/864 e 0 to 0 dl 1746323933 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 15950:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xf4:0x0] with magic=0xbd60bd0
traps: 11[19017] trap invalid opcode ip:404a6d sp:7e5f3468 error:0 in 11[400000+6000]
4[20229]: segfault at 8 ip 00007f3966fba7e8 sp 00007ffd727635c0 error 4 in ld-2.17.so[7f3966faf000+22000]
Lustre: 20277:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 20277:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20277:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20277:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 20277:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20277:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9497:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9497:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9497:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 9497:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9497:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9497:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15771:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15771:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15771:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15771:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15771:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15771:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15771:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 20277:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20277:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 20277:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 20277:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20277:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 20277:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20277:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9495:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9495:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9495:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 9495:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9495:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9495:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3 previous similar messages
LustreError: 5191:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800aa5f4a88: inode [0x200000402:0x734:0x0] mdc close failed: rc = -13
6[5824]: segfault at 0 ip 0000000000403cf0 sp 00007ffd7143ae18 error 6 in 6[400000+6000]
LustreError: 8161:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee7c37e8: inode [0x200000402:0x805:0x0] mdc close failed: rc = -13
0[9625]: segfault at ffffffffd9bd451f ip 0000000000404a87 sp 00007ffed9052ae8 error 7 in 0[400000+6000]
Lustre: 17519:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x8e2:0x0] with magic=0xbd60bd0
Lustre: 17519:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 9497:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9497:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 33 previous similar messages
Lustre: 9497:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9497:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 9497:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9497:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 9497:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9497:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 9497:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9497:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 9497:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9497:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 33 previous similar messages
LustreError: 12658:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802ee7c37e8: inode [0x200000401:0xa41:0x0] mdc close failed: rc = -13
2[14330]: segfault at 406000 ip 0000000000406000 sp 00007ffe77f10c18 error 14 in 2[606000+1000]
Lustre: 15170:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xb88:0x0] with magic=0xbd60bd0
Lustre: 15170:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
12[17223]: segfault at 8 ip 00007fc50c2727e8 sp 00007fff3e5e6bc0 error 4 in ld-2.17.so[7fc50c267000+22000]
6[17594]: segfault at 8 ip 00007fe7fd3467e8 sp 00007ffd3c873970 error 4 in ld-2.17.so[7fe7fd33b000+22000]
3[18239]: segfault at 8 ip 00007f0e915387e8 sp 00007ffcc7437250 error 4 in ld-2.17.so[7f0e9152d000+22000]
19[18571]: segfault at 8 ip 00007fd8a8f7a7e8 sp 00007fffe94007f0 error 4 in ld-2.17.so[7fd8a8f6f000+22000]
Lustre: 13564:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0xba8:0x0] with magic=0xbd60bd0
Lustre: 13564:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 14972:0:(lod_lov.c:1414:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xd26:0x0] with magic=0xbd60bd0
Lustre: 14972:0:(lod_lov.c:1414:lod_parse_striping()) Skipped 1 previous similar message
6[23784]: segfault at 8 ip 00007f59787ca7e8 sp 00007ffe72d57800 error 4 in ld-2.17.so[7f59787bf000+22000]
Lustre: 9496:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 618, rollback = 7
Lustre: 9496:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 9496:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9496:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 9496:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9496:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 9496:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 9496:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 9496:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9496:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 9496:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9496:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 39 previous similar messages
8[27430]: segfault at 0 ip 0000000000403e5f sp 00007ffe04bd6c30 error 6 in 13[400000+6000]
Lustre: mdt00_000: service thread pid 8707 was inactive for 40.100 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: mdt00_001: service thread pid 8709 was inactive for 40.104 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 8709, comm: mdt00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 13564, comm: mdt00_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_object_find_lock+0x54/0x170 [mdt]
[<0>] mdt_reint_setxattr+0x133/0x10e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_011: service thread pid 17521 was inactive for 40.101 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Pid: 8707, comm: mdt00_000 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 8525:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8800a544a5c0/0x7f1562181356b3f4 lrc: 3/0,0 mode: CR/CR res: [0x200000401:0xf5a:0x0].0x0 bits 0xa/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x7f1562181356b3e6 expref: 619 pid: 14947 timeout: 462 lvb_type: 0
Lustre: mdt00_016: service thread pid 13564 completed after 100.298s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 8709 completed after 100.293s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800aa5f4a88: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8800aa5f4a88: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff8800aa5f4a88: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 29782:0:(file.c:6143:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0xf5a:0x0] error: rc = -5
LustreError: 15174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746324270 with bad export cookie 9157333272868661088
LustreError: 30163:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800aa5f4a88: inode [0x200000401:0xf01:0x0] mdc close failed: rc = -108
Lustre: mdt00_011: service thread pid 17521 completed after 100.312s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 8707:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a9daefc8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802db8ad680/0x7f1562181356b608 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0xf5a:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x7f1562181356b5ec expref: 4 pid: 8707 timeout: 0 lvb_type: 0
Lustre: mdt00_005: service thread pid 14972 completed after 100.175s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 8707 completed after 100.316s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 29886:0:(vvp_io.c:1905:vvp_io_init()) lustre: refresh file layout [0x200000401:0xf5a:0x0] error -108.
Lustre: lustre-MDT0000-mdc-ffff8800aa5f4a88: Connection restored to 192.168.123.2@tcp (at 0@lo)
Lustre: 15771:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 15771:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 15771:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15771:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15771:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15771:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15771:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 15771:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15771:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15771:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15771:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15771:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3 previous similar messages
0[30453]: segfault at 8 ip 00007f4494ede7e8 sp 00007ffd0eb89410 error 4 in ld-2.17.so[7f4494ed3000+22000]
5[31451]: segfault at 0 ip 0000000000403e5f sp 00007ffeabc86350 error 6 in 5[400000+6000]
5[31837]: segfault at 8 ip 00007f3b689017e8 sp 00007ffe59a74050 error 4 in ld-2.17.so[7f3b688f6000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
PGD 9d34f067 PUD 9d2b1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel squashfs aesni_intel lrw gf128mul i2c_piix4 glue_helper i2c_core ablk_helper cryptd pcspkr binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi ata_piix crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libata
CPU: 0 PID: 2249 Comm: ll_sa_32447 Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
task: ffff88009d2a4440 ti: ffff88008d1c8000 task.ti: ffff88008d1c8000
RIP: 0010:[<ffffffff813f40ac>] [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88008d1cbd78 EFLAGS: 00010296
RAX: ffff8800849d08d0 RBX: 0000000000000008 RCX: ffffea0004a9b100
RDX: 0000000000000006 RSI: ffff8800849d0c00 RDI: 0000000000000008
RBP: ffff88008d1cbd88 R08: ffff8800a44a9000 R09: 000000018010000f
R10: 0000000000000001 R11: ffff8800a44a9000 R12: 0000000000000000
R13: ffff88009cb43330 R14: ffff8800849d0c00 R15: ffff8800a44a9000
FS: 0000000000000000(0000) GS:ffff88013e200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000008814c000 CR4: 0000000000160ff0
Call Trace:
[<ffffffffa10dbc41>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa10dae90>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817f3e5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

ln (11093) used greatest stack depth: 10320 bytes left
random: crng init done
18[11913]: segfault at 8 ip 00007fa30ac7e7e8 sp 00007ffd453fec40 error 4 in ld-2.17.so[7fa30ac73000+22000]
LustreError: 11789:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x240000403:0xb:0x0] mdc close failed: rc = -13
1[13511]: segfault at 8 ip 00007f12c0c0f7e8 sp 00007ffc11f549b0 error 4 in ld-2.17.so[7f12c0c04000+22000]
LustreError: 10899:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x95:0x0]: rc = -5
LustreError: 10899:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 13882:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x200000402:0xd:0x0] mdc close failed: rc = -13
LustreError: 14063:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x93:0x0]: rc = -5
LustreError: 14063:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
8[14509]: segfault at 0 ip 00000000004043f0 sp 00007fffb4729eb8 error 6 in 8[400000+6000]
LustreError: 17834:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x175:0x0]: rc = -5
LustreError: 17834:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 17834:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 17834:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 16832:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x240000403:0x175:0x0] : rc = -5
LustreError: 16832:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x175:0x0] error -5.
Lustre: dir [0x240000403:0x14e:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 19035:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x228:0x0]: rc = -5
LustreError: 19035:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 19035:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 19035:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 4 previous similar messages
LustreError: 20035:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x200000403:0x2fa:0x0] mdc close failed: rc = -2
Lustre: dir [0x200000403:0x35d:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 22468:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x32c:0x0]: rc = -5
LustreError: 22468:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 24289:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012ea9c800: cannot apply new layout on [0x200000402:0x32c:0x0] : rc = -5
LustreError: 24289:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000402:0x32c:0x0] error -5.
Lustre: dir [0x200000402:0x559:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 27764:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x200000403:0x59e:0x0] get parent: rc = -116
LustreError: 28141:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x240000403:0x479:0x0] mdc close failed: rc = -2
Lustre: dir [0x200000402:0x71d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 29865:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x4cf:0x0]: rc = -5
LustreError: 29865:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 11 previous similar messages
LustreError: 29865:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 29865:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 11 previous similar messages
LustreError: 29242:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x200000403:0x462:0x0] : rc = -5
LustreError: 29242:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000403:0x462:0x0] error -5.
LustreError: 32107:0:(llite_nfs.c:430:ll_dir_get_parent_fid()) lustre: failure inode [0x240000402:0x7f4:0x0] get parent: rc = -116
LustreError: 10761:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x200000403:0x462:0x0] : rc = -5
LustreError: 10761:0:(lov_object.c:1341:lov_layout_change()) Skipped 3 previous similar messages
8[2887]: segfault at 0 ip (null) sp 00007ffdc3a8f6d8 error 14 in 8[400000+6000]
Lustre: dir [0x240000402:0x8ee:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
1[8838]: segfault at 0 ip 0000000000403e5f sp 00007ffef8dc5870 error 6 in 1[400000+6000]
LustreError: 10976:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xae0:0x0]: rc = -5
LustreError: 10976:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 149 previous similar messages
LustreError: 10976:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 10976:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 149 previous similar messages
LustreError: 10935:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x240000403:0x792:0x0] mdc close failed: rc = -2
LustreError: 10935:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x200000403:0xd37:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
13[13893]: segfault at 8 ip 00007f97910db7e8 sp 00007ffd9b1e4ea0 error 4 in ld-2.17.so[7f97910d0000+22000]
LustreError: 487:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 17 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 487:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 7 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
17[14813]: segfault at 0 ip 0000000000403e5f sp 00007ffd8d03d020 error 6 in 17[400000+6000]
14[15287]: segfault at 8 ip 00007f38fb29f7e8 sp 00007fffbbed8eb0 error 4 in ld-2.17.so[7f38fb294000+22000]
LustreError: 19363:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012ea9c800: inode [0x200000403:0xee0:0x0] mdc close failed: rc = -2
LustreError: 19141:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x200000403:0x1087:0x0] : rc = -5
LustreError: 19141:0:(lov_object.c:1341:lov_layout_change()) Skipped 1 previous similar message
LustreError: 19141:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1087:0x0] error -5.
LustreError: 19141:0:(vvp_io.c:1903:vvp_io_init()) Skipped 1 previous similar message
14[24264]: segfault at 8 ip 00007fcf129677e8 sp 00007ffe64192180 error 4 in ld-2.17.so[7fcf1295c000+22000]
11[26163]: segfault at 8 ip 00007f03beb447e8 sp 00007ffc33008790 error 4 in ld-2.17.so[7f03beb39000+22000]
3[26486]: segfault at 0 ip (null) sp 00007ffc3e0fc908 error 14 in 3[400000+6000]
LustreError: 27579:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x240000403:0x753:0x0] : rc = -5
LustreError: 27579:0:(lov_object.c:1341:lov_layout_change()) Skipped 1 previous similar message
LustreError: 27579:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x753:0x0] error -5.
LustreError: 27579:0:(vvp_io.c:1903:vvp_io_init()) Skipped 1 previous similar message
14[29188]: segfault at 8 ip 00007fde0d28f7e8 sp 00007ffdaa1e9d60 error 4 in ld-2.17.so[7fde0d284000+22000]
6[29171]: segfault at 8 ip 00007f65241d77e8 sp 00007ffc49437eb0 error 4 in ld-2.17.so[7f65241cc000+22000]
13[31420]: segfault at 0 ip 0000000000403e5f sp 00007ffdefe334e0 error 6 in 13[400000+6000]
Lustre: dir [0x240000402:0x14f7:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
9[3297]: segfault at 8 ip 00007f98905657e8 sp 00007ffc0ed84850 error 4 in ld-2.17.so[7f989055a000+22000]
LustreError: 2497:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x240000402:0xafe:0x0] mdc close failed: rc = -2
LustreError: 2497:0:(file.c:247:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 4863:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000402:0x17dc:0x0] error -5.
16[5812]: segfault at 8 ip 00007fc6f0e837e8 sp 00007ffedb44aa90 error 4 in ld-2.17.so[7fc6f0e78000+22000]
2[6832]: segfault at 8 ip 00007fa5ddcb07e8 sp 00007fff21272f50 error 4 in ld-2.17.so[7fa5ddca5000+22000]
9[7147]: segfault at 8 ip 00007f544320f7e8 sp 00007fff82fef650 error 4 in ld-2.17.so[7f5443204000+22000]
LustreError: 6818:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x1817:0x0]: rc = -5
LustreError: 6818:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 97 previous similar messages
LustreError: 6818:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 6818:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 97 previous similar messages
LustreError: 14052:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012ea9c800: cannot apply new layout on [0x200000402:0x17dc:0x0] : rc = -5
LustreError: 14052:0:(lov_object.c:1341:lov_layout_change()) Skipped 3 previous similar messages
8[15074]: segfault at 8 ip 00007f7a3448b7e8 sp 00007fff5e788580 error 4 in ld-2.17.so[7f7a34480000+22000]
18[16759]: segfault at 406000 ip 0000000000406000 sp 00007ffefa797d98 error 14 in 18[606000+1000]
19[18967]: segfault at 8 ip 00007f55ef9f77e8 sp 00007ffe6f8b7ec0 error 4 in ld-2.17.so[7f55ef9ec000+22000]
LustreError: 23661:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x1c1f:0x0] error -5.
LustreError: 23661:0:(vvp_io.c:1903:vvp_io_init()) Skipped 4 previous similar messages
4[28247]: segfault at 0 ip (null) sp 00007fff272bbd58 error 14 in 4[400000+6000]
15[29010]: segfault at 8 ip 00007f77cd5d67e8 sp 00007ffcb28f5ef0 error 4 in ld-2.17.so[7f77cd5cb000+22000]
3[29880]: segfault at 0 ip 0000000000403e5f sp 00007ffd492433a0 error 6 in 16[400000+6000]
Lustre: dir [0x240000402:0x1f95:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 13 previous similar messages
LustreError: 30596:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012ea9c800: inode [0x200000403:0x2052:0x0] mdc close failed: rc = -2
LustreError: 30596:0:(file.c:247:ll_close_inode_openhandle()) Skipped 10 previous similar messages
LustreError: 1162:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 15 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 1162:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 2 previous similar messages
4[4769]: segfault at 8 ip 00007ff02e3b27e8 sp 00007ffffba02170 error 4 in ld-2.17.so[7ff02e3a7000+22000]
4[5197]: segfault at 8 ip 00007f45d721d7e8 sp 00007fff27fa7860 error 4 in ld-2.17.so[7f45d7212000+22000]
14[7446]: segfault at 8 ip 00007f539cd817e8 sp 00007ffea4e204f0 error 4 in ld-2.17.so[7f539cd76000+22000]
LustreError: 9753:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012ea9c800: cannot apply new layout on [0x240000402:0x2041:0x0] : rc = -5
LustreError: 9753:0:(lov_object.c:1341:lov_layout_change()) Skipped 39 previous similar messages
1[12262]: segfault at 8 ip 00007feec79447e8 sp 00007fff557a3e80 error 4 in ld-2.17.so[7feec7939000+22000]
0[19731]: segfault at 0 ip 00000000004016cd sp 00007fff82a10d90 error 6 in 0[400000+6000]
14[23596]: segfault at 406000 ip 0000000000406000 sp 00007ffcf52aca68 error 14 in 14[606000+1000]
LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 19 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 24827:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000402:0x21ef:0x0] error -5.
LustreError: 24827:0:(vvp_io.c:1903:vvp_io_init()) Skipped 13 previous similar messages
LustreError: 10495:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x2a2e:0x0]: rc = -5
LustreError: 10721:0:(llite_lib.c:3697:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 10721:0:(llite_lib.c:3697:ll_prep_inode()) Skipped 270 previous similar messages
LustreError: 10495:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 271 previous similar messages
17[731]: segfault at 0 ip 0000000000403e5f sp 00007ffc4df67140 error 6 in 17[400000+6000]
8[3969]: segfault at 0 ip 0000000000403e5f sp 00007ffd1c4721a0 error 6 in 8[400000+6000]
13[7101]: segfault at 8 ip 00007f371dee67e8 sp 00007ffc71e1e090 error 4 in ld-2.17.so[7f371dedb000+22000]
2[12229]: segfault at 8 ip 00007f0f28ba67e8 sp 00007ffd7f449470 error 4 in ld-2.17.so[7f0f28b9b000+22000]
0[14964]: segfault at 0 ip 0000000000401d60 sp 00007ffdf42c5568 error 6 in 0[400000+6000]
LustreError: 54:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 4 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
19[16730]: segfault at 8 ip 00007fec6960a7e8 sp 00007ffcd61b21d0 error 4 in ld-2.17.so[7fec695ff000+22000]
12[17198]: segfault at 8 ip 00007fbff82f77e8 sp 00007ffc8b50d700 error 4 in ld-2.17.so[7fbff82ec000+22000]
17[20432]: segfault at 8 ip 00007f662fe267e8 sp 00007ffdb781cca0 error 4 in ld-2.17.so[7f662fe1b000+22000]
Lustre: dir [0x240000402:0x3314:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 21 previous similar messages
16[25001]: segfault at 8 ip 00007f0d76e337e8 sp 00007ffdcbf894e0 error 4 in ld-2.17.so[7f0d76e28000+22000]
6[24946]: segfault at 0 ip 0000000000403e5f sp 00007ffe2489ff40 error 6 in 6[400000+6000]
LustreError: 26938:0:(llite_lib.c:1845:ll_update_lsm_md()) lustre: [0x200000403:0x33e5:0x0] dir layout mismatch:
LustreError: 26938:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 26938:0:(lustre_lmv.h:167:lmv_stripe_object_dump()) stripe[0] [0x200000400:0xb1:0x0]
LustreError: 26938:0:(lustre_lmv.h:160:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 32329:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012a528000: inode [0x200000402:0x3cba:0x0] mdc close failed: rc = -2
LustreError: 32329:0:(file.c:247:ll_close_inode_openhandle()) Skipped 9 previous similar messages
LustreError: lustre-OST0000-osc-ffff88012a528000: operation ldlm_enqueue to node 192.168.203.144@tcp failed: rc = -107
Lustre: lustre-OST0000-osc-ffff88012a528000: Connection to lustre-OST0000 (at 192.168.203.144@tcp) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 2014:0:(osc_request.c:1005:osc_init_grant()) lustre-OST0000-osc-ffff88012a528000: granted 8437760 but already consumed 23216128
LustreError: lustre-OST0000-osc-ffff88012a528000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x3ec1:0x0]/ may get corrupted (rc -5)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x34fa:0x0]// may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x3f16:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x3f27:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000403:0x3cbe:0x0]/ may get corrupted (rc -108)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000403:0x352d:0x0]/ may get corrupted (rc -5)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000403:0x3c88:0x0]/ may get corrupted (rc -5)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x3672:0x0]// may get corrupted (rc -108)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000403:0x30c9:0x0]/ may get corrupted (rc -5)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x2db6:0x0]/ may get corrupted (rc -5)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x3632:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000403:0x368c:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x3f42:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x3530:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x3fac:0x0]// may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x368f:0x0]// may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x3694:0x0]/ may get corrupted (rc -108)
Lustre: 2018:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000403:0x3d0b:0x0]/ may get corrupted (rc -108)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000403:0x1330:0x0]// may get corrupted (rc -5)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000402:0x2fac:0x0]/ may get corrupted (rc -5)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x200000402:0x197f:0x0]/ may get corrupted (rc -5)
Lustre: 2017:0:(llite_lib.c:4142:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.203.144@tcp:/lustre/fid: [0x240000403:0x2919:0x0]/ may get corrupted (rc -5)
LustreError: 3132:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-OST0000-osc-ffff88012a528000: namespace resource [0x280000400:0x822:0x0].0x0 (ffff8800a5940900) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff88012a528000: Connection restored to (at 192.168.203.144@tcp)
LustreError: 3528:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012a528000: cannot apply new layout on [0x240000403:0x3330:0x0] : rc = -5
LustreError: 3528:0:(lov_object.c:1341:lov_layout_change()) Skipped 11 previous similar messages
4[4834]: segfault at 8 ip 00007f39a51aa7e8 sp 00007ffef22a83c0 error 4 in ld-2.17.so[7f39a519f000+22000]
14[5638]: segfault at 8 ip 00007f5652c107e8 sp 00007fff2c2b4cf0 error 4 in ld-2.17.so[7f5652c05000+22000]
16[10341]: segfault at 8 ip 00007f78ac1e27e8 sp 00007ffea71b9eb0 error 4 in ld-2.17.so[7f78ac1d7000+22000]
LustreError: 20627:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000402:0x3bb2:0x0] error -5.
LustreError: 20627:0:(vvp_io.c:1903:vvp_io_init()) Skipped 6 previous similar messages
16[21717]: segfault at 8 ip 00007fcf216f77e8 sp 00007ffc44d15c20 error 4 in ld-2.17.so[7fcf216ec000+22000]
8[23663]: segfault at 8 ip 00007f64367ee7e8 sp 00007fff388d0e10 error 4 in ld-2.17.so[7f64367e3000+22000]
LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 18 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
2[27154]: segfault at 8 ip 00007f40385957e8 sp 00007ffff80c0b30 error 4 in ld-2.17.so[7f403858a000+22000]
2[27626]: segfault at 8 ip 00007f0c67b437e8 sp 00007ffcc1473710 error 4 in ld-2.17.so[7f0c67b38000+22000]
0[27982]: segfault at 8 ip 00007f8b3039f7e8 sp 00007fff4b0bca70 error 4 in ld-2.17.so[7f8b30394000+22000]
3[28859]: segfault at 8 ip 00007fb2c995a7e8 sp 00007ffd05ae6210 error 4 in ld-2.17.so[7fb2c994f000+22000]
0[29844]: segfault at 8 ip 00007f563f15e7e8 sp 00007fff72ebddc0 error 4 in ld-2.17.so[7f563f153000+22000]
0[32049]: segfault at 0 ip 0000000000403e5f sp 00007ffc8610aa60 error 6 in 0[400000+6000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 2ca984067 PUD 2687f4067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix virtio_blk drm_panel_orientation_quirks serio_raw floppy libata i2c_core
CPU: 13 PID: 30928 Comm: ll_sa_30892 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880091868010 ti: ffff8802804c4000 task.ti: ffff8802804c4000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802804c7d78 EFLAGS: 00010296
RAX: ffff88026d88d690 RBX: 0000000000000008 RCX: 000000010026001d
RDX: 0000000000000026 RSI: ffff88026d88d9c0 RDI: 0000000000000008
RBP: ffff8802804c7d88 R08: ffff880262564f88 R09: 0000000000000000
R10: ffff8802625648e8 R11: ffff880262564f88 R12: 0000000000000000
R13: ffff8800972d8010 R14: ffff88026d88d9c0 R15: ffff880262564f88
FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000025fe02000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15b3bf1>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff817e320e>] ? _raw_spin_lock+0x1e/0x20
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa15b2e40>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

11[25698]: segfault at 0 ip (null) sp 00007fffbed052e8 error 14 in 11[400000+6000]
Lustre: 24186:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d6f87340 x1830163809158144/t4294969685(0) o101->5d557860-6452-4659-85e9-4540b219faa6@0@lo:5/0 lens 376/816 e 0 to 0 dl 1745380315 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 24186:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xc8:0x0] with magic=0xbd60bd0
15[30936]: segfault at 8 ip 00007fdd73cf87e8 sp 00007fff7fabf2d0 error 4 in ld-2.17.so[7fdd73ced000+22000]
LustreError: 8084:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a2eb4138: inode [0x200000401:0x269:0x0] mdc close failed: rc = -13
Lustre: 22371:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x3c9:0x0] with magic=0xbd60bd0
Lustre: 22371:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 29058:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a2eb4138: inode [0x200000402:0x59a:0x0] mdc close failed: rc = -13
19[30849]: segfault at 8 ip 00007f1cfba507e8 sp 00007fff5c014370 error 4 in ld-2.17.so[7f1cfba45000+22000]
Lustre: 23071:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x771:0x0] with magic=0xbd60bd0
Lustre: 23071:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 11845:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x768:0x0] with magic=0xbd60bd0
Lustre: 11845:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
4[11357]: segfault at 8 ip 00007fd7720007e8 sp 00007ffcce67f290 error 4 in ld-2.17.so[7fd771ff5000+22000]
14[14444]: segfault at 8 ip 00007f7948d4f7e8 sp 00007fff8a0c0fe0 error 4 in ld-2.17.so[7f7948d44000+22000]
18[16343]: segfault at 0 ip 0000000000403e5f sp 00007ffdf694ea80 error 6 in 18[400000+6000]
LustreError: 25744:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a2eb4138: inode [0x200000401:0x966:0x0] mdc close failed: rc = -13
Lustre: 2992:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x9da:0x0] with magic=0xbd60bd0
Lustre: 2992:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 22287:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x946:0x0] ACL: rc = -2
Lustre: 11844:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0xfc8:0x0] with magic=0xbd60bd0
Lustre: 11844:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 2995:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x10d3:0x0] with magic=0xbd60bd0
Lustre: 2995:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 2995:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x10f6:0x0] ACL: rc = -2
13[25983]: segfault at 8 ip 00007f87a96517e8 sp 00007ffc701e0300 error 4 in ld-2.17.so[7f87a9646000+22000]
LustreError: 13400:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802de7e4a88: inode [0x200000402:0x1515:0x0] mdc close failed: rc = -13
Lustre: 25147:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000402:0x172c:0x0] with magic=0xbd60bd0
Lustre: 25147:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 4613:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802de7e4a88: inode [0x200000401:0x186e:0x0] mdc close failed: rc = -13
5[22635]: segfault at 8 ip 00007fb16b8c37e8 sp 00007ffd6becc4c0 error 4 in ld-2.17.so[7fb16b8b8000+22000]
LustreError: 2997:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000402:0x1b38:0x0] ACL: rc = -2
LustreError: 2997:0:(mdt_handler.c:746:mdt_pack_acl2body()) Skipped 1 previous similar message
LustreError: 17380:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802de7e4a88: inode [0x200000401:0x1e95:0x0] mdc close failed: rc = -13
LustreError: 17380:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
9[19198]: segfault at 8 ip 00007f2e2ad2a7e8 sp 00007fff1c7c6fd0 error 4 in ld-2.17.so[7f2e2ad1f000+22000]
Lustre: 22514:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x1e8d:0x0] with magic=0xbd60bd0
Lustre: 22514:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 21445:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802de7e4a88: inode [0x200000401:0x2360:0x0] mdc close failed: rc = -13
LustreError: 21445:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
14[27964]: segfault at 8 ip 00007f693d13a7e8 sp 00007ffef9a7e3a0 error 4 in ld-2.17.so[7f693d12f000+22000]
Lustre: 14278:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000401:0x2410:0x0] with magic=0xbd60bd0
Lustre: 14278:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 3 previous similar messages
9[29671]: segfault at 0 ip 0000000000403e5f sp 00007ffca0d2f730 error 6 in 9[400000+6000]
LustreError: 14278:0:(mdt_handler.c:746:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x24dc:0x0] ACL: rc = -2
17[10172]: segfault at 8 ip 00007fb8b56ee7e8 sp 00007ffce936b430 error 4 in ld-2.17.so[7fb8b56e3000+22000]
17[10701]: segfault at 8 ip 00007fcdddbfa7e8 sp 00007ffdf2249850 error 4 in ld-2.17.so[7fcdddbef000+22000]
LustreError: 14633:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a2eb4138: inode [0x200000402:0x25fa:0x0] mdc close failed: rc = -13
0[20514]: segfault at 8 ip 00007fb74fb237e8 sp 00007fffe1b56a30 error 4 in ld-2.17.so[7fb74fb18000+22000]

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
PGD 12d51e067 PUD 136521067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rpcsec_gss_krb5 sb_edac edac_core iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel squashfs aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 pcspkr i2c_core binfmt_misc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw libata
CPU: 2 PID: 27412 Comm: ll_sa_24787 Kdump: loaded Tainted: G OE ------------ 3.10.0-7.9-debug #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
task: ffff88009db80000 ti: ffff88008bad8000 task.ti: ffff88008bad8000
RIP: 0010:[<ffffffff813f40ac>] [<ffffffff813f40ac>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff88008badbd78 EFLAGS: 00010296
RAX: ffff88008b19a8d0 RBX: 0000000000000008 RCX: 000000018010000e
RDX: 000000018010000f RSI: ffff88008b19ac00 RDI: 0000000000000008
RBP: ffff88008badbd88 R08: ffff88009421e600 R09: 000000018010000e
R10: 000000009421fe01 R11: ffff88009421e600 R12: 0000000000000000
R13: ffff8800a4c16660 R14: ffff88008b19ac00 R15: ffff88009421e600
FS: 0000000000000000(0000) GS:ffff88013e300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000130e56000 CR4: 0000000000160fe0
Call Trace:
[<ffffffffa10d3c51>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff81410a79>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e8dce>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa10d2ea0>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817f3e5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

cat (11140) used greatest stack depth: 10608 bytes left
random: crng init done
LustreError: 12268:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012bf2e800: inode [0x200000402:0x2c:0x0] mdc close failed: rc = -116
LustreError: 12563:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012bf2e800: inode [0x200000402:0x57:0x0] mdc close failed: rc = -116
LustreError: 12563:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 13796:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b6d34800: inode [0x240000402:0x39:0x0] mdc close failed: rc = -116
LustreError: 13796:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x200000403:0x122:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
10[15435]: segfault at 0 ip 0000000000403e5f sp 00007ffde1b08de0 error 6 in 10[400000+6000]
LustreError: 16093:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b6d34800: inode [0x200000403:0x152:0x0] mdc close failed: rc = -116
LustreError: 16093:0:(file.c:247:ll_close_inode_openhandle()) Skipped 3 previous similar messages
LustreError: 16179:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x95:0x0]: rc = -5
LustreError: 16179:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
Lustre: dir [0x200000403:0x125:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
4[17052]: segfault at 0 ip 0000000000403e5f sp 00007ffd225a4930 error 6 in 4[400000+6000]
LustreError: 17629:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x95:0x0]: rc = -5
LustreError: 17629:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 17598:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x240000402:0x1be:0x0] : rc = -5
LustreError: 17598:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000402:0x1be:0x0] error -5.
LustreError: 16862:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b6d34800: inode [0x200000402:0x1a3:0x0] mdc close failed: rc = -2
LustreError: 16862:0:(file.c:247:ll_close_inode_openhandle()) Skipped 15 previous similar messages
LustreError: 19634:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x240000402:0x95:0x0] : rc = -5
LustreError: 19634:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000402:0x95:0x0] error -5.
14[22218]: segfault at 8 ip 00007fd9c8f107e8 sp 00007ffd834b7da0 error 4 in ld-2.17.so[7fd9c8f05000+22000]
LustreError: 22372:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x200000402:0x359:0x0] : rc = -5
LustreError: 22372:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000402:0x359:0x0] error -5.
Lustre: dir [0x240000402:0x2db:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
17[23726]: segfault at 8 ip 00007f1f6bd527e8 sp 00007ffc5be9a4b0 error 4 in ld-2.17.so[7f1f6bd47000+22000]
LustreError: 25039:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x29a:0x0]: rc = -5
LustreError: 25039:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 25039:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 25039:0:(llite_lib.c:3698:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: dir [0x240000403:0x3ca:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
LustreError: 25483:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x200000402:0x3de:0x0] : rc = -5
LustreError: 25483:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000402:0x3de:0x0] error -5.
LustreError: 26588:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x315:0x0]: rc = -5
LustreError: 26588:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 6 previous similar messages
LustreError: 26588:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 26588:0:(llite_lib.c:3698:ll_prep_inode()) Skipped 6 previous similar messages
LustreError: 27481:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012bf2e800: inode [0x200000403:0x427:0x0] mdc close failed: rc = -116
LustreError: 27481:0:(file.c:247:ll_close_inode_openhandle()) Skipped 34 previous similar messages
3[28566]: segfault at 8 ip 00007f1420bf07e8 sp 00007ffe5e2e9370 error 4 in ld-2.17.so[7f1420be5000+22000]
19[29312]: segfault at 8 ip 00007f1fb32bd7e8 sp 00007ffcc1afaa90 error 4 in ld-2.17.so[7f1fb32b2000+22000]
LustreError: 31372:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000402:0x55d:0x0]: rc = -5
LustreError: 31372:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 31372:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 31372:0:(llite_lib.c:3698:ll_prep_inode()) Skipped 1 previous similar message
2[31258]: segfault at 8 ip 00007f2eb91f97e8 sp 00007fff7044e790 error 4 in ld-2.17.so[7f2eb91ee000+22000]
Lustre: lustre-MDT0000-mdc-ffff88012bf2e800: Connection to lustre-MDT0000 (at 192.168.204.147@tcp) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffff88012bf2e800: operation ldlm_enqueue to node 192.168.204.147@tcp failed: rc = -107
LustreError: lustre-MDT0000-mdc-ffff88012bf2e800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 28610:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff88012bf2e800: inode [0x200000403:0x427:0x0] mdc close failed: rc = -108
LustreError: 27080:0:(mdc_request.c:1464:mdc_read_page()) lustre-MDT0000-mdc-ffff88012bf2e800: [0x200000400:0x22:0x0] lock enqueue fails: rc = -5
Lustre: dir [0x200000403:0x5aa:0x0] stripe 0 readdir failed: -5, directory is partially accessed!
LustreError: 31411:0:(file.c:6138:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000402:0x7a9:0x0] error: rc = -5
LustreError: 28610:0:(file.c:247:ll_close_inode_openhandle()) Skipped 9 previous similar messages
LustreError: 31669:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88012bf2e800: namespace resource [0x200000007:0x1:0x0].0x0 (ffff88009dbe8600) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 32203:0:(file.c:6138:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 32203:0:(file.c:6138:ll_inode_revalidate_fini()) Skipped 636 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88012bf2e800: Connection restored to (at 192.168.204.147@tcp)
9[412]: segfault at 8 ip 00007f7ac407d7e8 sp 00007ffe52934a50 error 4 in ld-2.17.so[7f7ac4072000+22000]
LustreError: 46:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000402:0x5cb:0x0]: rc = -5
LustreError: 46:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 46:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 46:0:(llite_lib.c:3698:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 46:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 9 [0x200000402:0x5cb:0x0] inode@0000000000000000: rc = -5
10[901]: segfault at 8 ip 00007f92eee8b7e8 sp 00007ffd4105cee0 error 4 in ld-2.17.so[7f92eee80000+22000]
19[3804]: segfault at 0 ip (null) sp 00007fffc75d04a8 error 14 in 19[400000+6000]
10[5782]: segfault at 1c ip 00000000004017b0 sp 00007ffc1e51cfa0 error 6 in 10[400000+6000]
Lustre: dir [0x240000402:0x355:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
16[7752]: segfault at 8 ip 00007f5dfbda87e8 sp 00007ffcce401230 error 4 in ld-2.17.so[7f5dfbd9d000+22000]
6[7833]: segfault at 8 ip 00007f0f4ffc97e8 sp 00007ffec5a8f6e0 error 4 in ld-2.17.so[7f0f4ffbe000+22000]
3[8062]: segfault at 8 ip 00007ff1494907e8 sp 00007ffcad45b9f0 error 4 in ld-2.17.so[7ff149485000+22000]
9[12135]: segfault at 8 ip 00007f42ecc317e8 sp 00007ffc08000d60 error 4 in ld-2.17.so[7f42ecc26000+22000]
LustreError: 11615:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x240000402:0xb8b:0x0] : rc = -5
LustreError: 11615:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000402:0xb8b:0x0] error -5.
4[13826]: segfault at 8 ip 00007f23687297e8 sp 00007ffc248a3df0 error 4 in ld-2.17.so[7f236871e000+22000]
LustreError: 11838:0:(lcommon_cl.c:179:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xa0a:0x0]: rc = -5
LustreError: 11838:0:(lcommon_cl.c:179:cl_file_inode_init()) Skipped 56 previous similar messages
LustreError: 11838:0:(llite_lib.c:3698:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 11838:0:(llite_lib.c:3698:ll_prep_inode()) Skipped 56 previous similar messages
5[15028]: segfault at 406000 ip 0000000000406000 sp 00007ffd92c15418 error 14 in 5[606000+1000]
8[16414]: segfault at 0 ip 0000000000403e5f sp 00007ffc1830cd10 error 6 in 8[400000+6000]
2[18340]: segfault at 8 ip 00007ff9dabde7e8 sp 00007ffcedd896b0 error 4 in ld-2.17.so[7ff9dabd3000+22000]
2[18399]: segfault at 8 ip 00007f2825fad7e8 sp 00007ffdfade06d0 error 4 in ld-2.17.so[7f2825fa2000+22000]
18[18681]: segfault at 8 ip 00007f2fa1e5e7e8 sp 00007ffd471bdd90 error 4 in ld-2.17.so[7f2fa1e53000+22000]
8[20673]: segfault at 8 ip 00007f8658e5f7e8 sp 00007ffefe0e8db0 error 4 in ld-2.17.so[7f8658e54000+22000]
Lustre: dir [0x240000403:0xdca:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 20731:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff88012bf2e800: cannot apply new layout on [0x200000402:0x96c:0x0] : rc = -5
LustreError: 20731:0:(lov_object.c:1341:lov_layout_change()) Skipped 5 previous similar messages
LustreError: 24593:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000404:0x6e2:0x0] error -5.
LustreError: 24593:0:(vvp_io.c:1903:vvp_io_init()) Skipped 3 previous similar messages
LustreError: 26570:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800b6d34800: inode [0x200000404:0x9b9:0x0] mdc close failed: rc = -116
LustreError: 26570:0:(file.c:247:ll_close_inode_openhandle()) Skipped 70 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
PGD 289a07067 PUD 2c9ebc067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix serio_raw drm_panel_orientation_quirks virtio_blk i2c_core libata floppy
CPU: 3 PID: 22239 Comm: ll_sa_22015 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802d0bfdc40 ti: ffff8802bb0c0000 task.ti: ffff8802bb0c0000
RIP: 0010:[<ffffffff813f502c>] [<ffffffff813f502c>] _atomic_dec_and_lock+0xc/0x70
RSP: 0018:ffff8802bb0c3d78 EFLAGS: 00010296
RAX: ffff8802794d3090 RBX: 0000000000000008 RCX: 0000000100260022
RDX: 0000000000000026 RSI: ffff8802794d33c0 RDI: 0000000000000008
RBP: ffff8802bb0c3d88 R08: ffff8800a06b2128 R09: 0000000000000000
R10: ffff8800a06b52d8 R11: ffff8800a06b2128 R12: 0000000000000000
R13: ffff8802c12724f0 R14: ffff8802794d33c0 R15: ffff8800a06b2128
FS: 0000000000000000(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002bcf3e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1114461>] ll_statahead_thread+0xdb1/0x2000 [lustre]
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffffa11136b0>] ? ll_statahead_handle.constprop.28+0x1c0/0x1c0 [lustre]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 12468:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12468:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12468:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12468:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12468:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12468:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
traps: 0[18237] general protection ip:404b70 sp:7ffe1ec03168 error:0 in 9[400000+6000]
touch (17565) used greatest stack depth: 10080 bytes left
Lustre: 18501:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 511 < left 699, rollback = 2
Lustre: 18501:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0
Lustre: 18501:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18501:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 18501:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18501:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 1/3/0
Lustre: 18501:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18501:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 12/275/3, delete: 0/0/0
Lustre: 18501:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18501:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0
Lustre: 18501:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12470:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12470:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 12470:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12470:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 12470:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12470:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 12470:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12470:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 12470:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12470:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 12470:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12470:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 18501:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/0 is open, migrate only dentry
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 1 previous similar message
9[19215]: segfault at 8 ip 00007f657265f7e8 sp 00007ffcf3e1d040 error 4 in ld-2.17.so[7f6572654000+22000]
Lustre: 10851:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 699, rollback = 2
Lustre: 10851:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 10831:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xc:0x0] with magic=0xbd60bd0
7[19911]: segfault at 8 ip 00007fe7b7bd77e8 sp 00007ffceccd6640 error 4 in ld-2.17.so[7fe7b7bcc000+22000]
Lustre: 17903:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802d2b48f40 x1829451541470208/t4294968147(0) o101->4c8b5d62-e3d7-411b-8f5a-4e94e2e0e8e8@0@lo:238/0 lens 376/864 e 0 to 0 dl 1744701048 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: dir [0x200000403:0x48:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: 10848:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240000404:0x2a:0x0]/13 is open, migrate only dentry
Lustre: 17168:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802c105a340 x1829451542002048/t4294968218(0) o101->4c8b5d62-e3d7-411b-8f5a-4e94e2e0e8e8@0@lo:241/0 lens 376/864 e 0 to 0 dl 1744701051 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
4[20254]: segfault at 8 ip 00007f95a3ec57e8 sp 00007ffc4bfe4e00 error 4 in ld-2.17.so[7f95a3eba000+22000]
Lustre: 10848:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/17 is open, migrate only dentry
Lustre: 12468:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12468:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 20664:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 788, rollback = 2
Lustre: 20664:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 17343:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 17343:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 17343:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 17343:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 17343:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 17343:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 17343:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 17343:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 17343:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17343:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 12468:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12468:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages
8[24671]: segfault at 406000 ip 0000000000406000 sp 00007ffc6b50f738 error 14 in 8[606000+1000]
12[25544]: segfault at 0 ip 0000000000403e5f sp 00007fff57027260 error 6 in 12[400000+6000]
Lustre: 12469:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12469:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12469:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12469:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12469:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/594/0, punch: 0/0/0, quota 1/3/0
Lustre: 12469:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12469:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12469:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12469:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12469:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10851:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 509 < left 1695, rollback = 2
Lustre: 10851:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 28 previous similar messages
Lustre: 12909:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2dc:0x0] with magic=0xbd60bd0
Lustre: 12909:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18195:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18195:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 18195:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18195:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18195:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 18501:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/4 is open, migrate only dentry
Lustre: 18501:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 877, rollback = 2
Lustre: 18501:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 47 previous similar messages
Lustre: dir [0x200000403:0x475:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
10[12016]: segfault at 0 ip 00000000004011b0 sp 00007ffd63dfaff0 error 6 in 10[400000+6000]
Lustre: 10852:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 510 < left 1500, rollback = 2
Lustre: 10852:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 23 previous similar messages
3[13539]: segfault at 8 ip 00007f64255f17e8 sp 00007fff57907490 error 4 in ld-2.17.so[7f64255e6000+22000]
LustreError: 13695:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x51d:0x0]: rc = -5
LustreError: 13695:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
1[16911]: segfault at 0 ip 0000000000403e5f sp 00007ffe175476a0 error 6 in 1[400000+6000]
Lustre: 15348:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x56c:0x0] with magic=0xbd60bd0
Lustre: 15348:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 18090:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e6059e40 x1829451557099648/t4294979173(0) o101->67fa0521-646f-41f6-aaa0-db2cd751d950@0@lo:340/0 lens 376/816 e 0 to 0 dl 1744701150 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 17343:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 564, rollback = 7
Lustre: 17343:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1 previous similar message
INFO: task file_chmod.sh:16310 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chmod.sh D ffff8802e626b760 12696 16310 16270 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_mknod.sh:16312 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_mknod.sh D ffff8802dd558040 12832 16312 16270 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_delxattr.s:16355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_delxattr.s D ffff8802dce2ba00 12800 16355 16270 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_setxattr.s:16359 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_setxattr.s D ffff8802e4be8608 12832 16359 16270 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_chmod.sh:16362 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chmod.sh D ffff88031acb9900 12832 16362 16262 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_chmod.sh:16719 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chmod.sh D ffff88031fbb2980 12800 16719 16292 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_chown.sh:16733 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chown.sh D ffff8802dd55ca80 12832 16733 16274 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_chmod.sh:16734 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chmod.sh D ffff8802dcd890c0 11424 16734 16274 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_chmod.sh:16737 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_chmod.sh D ffff8800a50de340 12696 16737 16278 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
INFO: task file_mknod.sh:16739 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_mknod.sh D ffff8800a785b760 12800 16739 16278 0x00000080
Call Trace:
[<ffffffff817e19d9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df7ea>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff812512e3>] ? unlazy_walk+0x153/0x160
[<ffffffff817dfa1d>] mutex_lock+0x2d/0x40
[<ffffffff817d5b92>] lookup_slow+0x33/0xa7
[<ffffffff81256dee>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f176>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c1f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256e4b>] filename_lookup+0x2b/0xc0
[<ffffffff81258d47>] user_path_at_empty+0x67/0xc0
[<ffffffff81258db1>] user_path_at+0x11/0x20
[<ffffffff8124b733>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bafe>] SYSC_newstat+0x2e/0x60
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124bfce>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
17[24380]: segfault at 1c ip 00000000004017b0 sp 00007ffd57572d90 error 6 in 17[400000+6000]
15[25222]: segfault at 0 ip (null) sp 00007ffe10357f78 error 14 in 15[400000+6000]
9[26460]: segfault at 0 ip (null) sp 00007ffdb5bf2ca8 error 14 in 9[400000+6000]
Lustre: 12469:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12469:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 70 previous similar messages
Lustre: 12469:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12469:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 70 previous similar messages
Lustre: 12469:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12469:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 70 previous similar messages
Lustre: 12469:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12469:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 70 previous similar messages
Lustre: 12469:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12469:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 70 previous similar messages
Lustre: 18055:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x810:0x0] with magic=0xbd60bd0
Lustre: 18055:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 15449:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e739ca88: inode [0x200000404:0x543:0x0] mdc close failed: rc = -13
Lustre: 10848:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/0 is open, migrate only dentry
7[30845]: segfault at 1c ip 00000000004017b0 sp 00007ffe3bda4e60 error 6 in 7[400000+6000]
9[30815]: segfault at 8 ip 00007f7ca002f7e8 sp 00007ffdf4b50c30 error 4 in ld-2.17.so[7f7ca0024000+22000]
traps: 5[32570] trap invalid opcode ip:405352 sp:7ffe9d0d6af8 error:0 in 5[400000+6000]
Lustre: 18501:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/5 is open, migrate only dentry
Lustre: 18501:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 3 previous similar messages
Lustre: 10852:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 1100, rollback = 2
Lustre: 10852:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 38 previous similar messages
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7
Lustre: 18195:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: dir [0x280000404:0x501:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
1[11088]: segfault at 8 ip 00007f11a305c7e8 sp 00007fffa7fa9c30 error 4 in ld-2.17.so[7f11a3051000+22000]
traps: 7[12176] trap invalid opcode ip:405b8a sp:7ffd5c2090b8 error:0 in 7[400000+6000]
Lustre: 18501:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/3 is open, migrate only dentry
traps: 7[15124] trap invalid opcode ip:404aad sp:fdd7f468 error:0
traps: 7[15158] trap invalid opcode ip:404aad sp:c2091468 error:0
in 7[400000+6000]
in 7[400000+6000]
15[17575]: segfault at 8 ip 00007f96016ad7e8 sp 00007ffdbece8060 error 4 in ld-2.17.so[7f96016a2000+22000]
Lustre: 18501:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 501 < left 877, rollback = 2
Lustre: 18501:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 62 previous similar messages
4[16927]: segfault at 8 ip 00007f2433d9e7e8 sp 00007fff1f2f8190 error 4 in ld-2.17.so[7f2433d93000+22000]
17[20506]: segfault at 8 ip 00007f29021327e8 sp 00007ffeb1fd2af0 error 4 in ld-2.17.so[7f2902127000+22000]
3[20826]: segfault at 8 ip 00007fd8b2b327e8 sp 00007fff3b836fd0 error 4 in ld-2.17.so[7fd8b2b27000+22000]
LustreError: 17794:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x632:0x0]: rc = -2
LustreError: 19102:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x240000403:0x632:0x0] mdc close failed: rc = -2
9[21222]: segfault at 8 ip 00007f9d207457e8 sp 00007ffc24e37ae0 error 4 in ld-2.17.so[7f9d2073a000+22000]
Lustre: 19946:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0001: [0x240000403:0x1:0x0]/9 is open, migrate only dentry
Lustre: dir [0x240000404:0x709:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
14[21895]: segfault at 8 ip 00007f289f7177e8 sp 00007ffc855d3bc0 error 4 in ld-2.17.so[7f289f70c000+22000]
Lustre: 10452:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0000: [0x200000403:0x1:0x0]/13 is open, migrate only dentry
LustreError: 18859:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x240000404:0x5a2:0x0]/14 failed: rc = -2
Lustre: 10851:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x2:0x0]/4 is open, migrate only dentry
Lustre: 10851:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 12 previous similar messages
LustreError: 20961:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x280000403:0x41d:0x0]/15 failed: rc = -2
LustreError: 73:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x722:0x0]: rc = -5
LustreError: 73:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 73:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep sleep [0x240000403:0x722:0x0] inode@0000000000000000: rc = -5
6[23690]: segfault at 8 ip 00007f1d3a7ff7e8 sp 00007ffc5d528b50 error 4 in ld-2.17.so[7f1d3a7f4000+22000]
11[23429]: segfault at 8 ip 00007f2203f497e8 sp 00007ffffeb3e200 error 4 in ld-2.17.so[7f2203f3e000+22000]
LustreError: 10851:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/15 failed: rc = -2
5[24466]: segfault at 1c ip 00000000004017b0 sp 00007fffc95d1740 error 6 in 19[400000+6000]
Lustre: 17168:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x456:0x0] with magic=0xbd60bd0
Lustre: 17168:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 10851:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/16 failed: rc = -2
LustreError: 26929:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x456:0x0]: rc = -5
LustreError: 26929:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 26929:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 26929:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 27732:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x456:0x0]: rc = -5
LustreError: 27732:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 27732:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 27732:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 4 previous similar messages
LustreError: 25255:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x280000403:0x4de:0x0] : rc = -5
LustreError: 25255:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x280000403:0x4de:0x0] error -5.
LustreError: 13319:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x44a:0x0] migrate mdt count mismatch 2 != 1
Lustre: 18068:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0xb98:0x0] with magic=0xbd60bd0
Lustre: 18068:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 23015:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200000404:0xcb1:0x0]/20 failed: rc = -2
LustreError: 28470:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x456:0x0]: rc = -5
LustreError: 28470:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 9 previous similar messages
LustreError: 28470:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 28470:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 9 previous similar messages
Lustre: dir [0x200000404:0xcb1:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 22669:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0001: '16' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 16' to finish migration: rc = -1
LustreError: 29615:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x240000404:0x7ff:0x0] mdc close failed: rc = -116
LustreError: 30092:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x280000403:0x4de:0x0] : rc = -5
LustreError: 18501:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x41f:0x0]/9 failed: rc = -2
LustreError: 18501:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 2 previous similar messages
LustreError: 31831:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0xdeb:0x0]: rc = -5
LustreError: 31831:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 7 previous similar messages
LustreError: 31831:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 31831:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 7 previous similar messages
Lustre: 22763:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 3/12/3, destroy: 1/4/0
Lustre: 22763:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 595 previous similar messages
Lustre: 22763:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 28/2407/0
Lustre: 22763:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 595 previous similar messages
Lustre: 22763:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 5/55/0, punch: 0/0/0, quota 7/129/0
Lustre: 22763:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 595 previous similar messages
Lustre: 22763:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 14/287/3, delete: 2/5/0
Lustre: 22763:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 595 previous similar messages
Lustre: 22763:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 1/1/0
Lustre: 22763:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 595 previous similar messages
3[31488]: segfault at 8 ip 00007f1a0d3b27e8 sp 00007ffec590bea0 error 4 in ld-2.17.so[7f1a0d3a7000+22000]
LustreError: 17794:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x500:0x0]: rc = -2
LustreError: 26720:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x200000403:0x500:0x0] mdc close failed: rc = -2
2[32402]: segfault at 406000 ip 0000000000406000 sp 00007ffcdb319de8 error 14 in 2[606000+1000]
LustreError: 22763:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: '19' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 19' to finish migration: rc = -1
LustreError: 18197:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xe17:0x0]: rc = -2
LustreError: 31023:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e739ca88: inode [0x200000403:0xe17:0x0] mdc close failed: rc = -2
Lustre: 10848:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 499 < left 1678, rollback = 2
Lustre: 10848:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 752 previous similar messages
LustreError: 19721:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/1 failed: rc = -2
LustreError: 19721:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 6 previous similar messages
16[2640]: segfault at 8 ip 00007f09a18347e8 sp 00007ffe67947470 error 4 in ld-2.17.so[7f09a1829000+22000]
LustreError: 3528:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e739ca88: inode [0x200000403:0xeee:0x0] mdc close failed: rc = -13
LustreError: 3528:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 4419:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x519:0x0]: rc = -5
LustreError: 4419:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 18501:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 5' to finish migration: rc = -1
LustreError: 3502:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x240000403:0x9d9:0x0] : rc = -5
LustreError: 3502:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x9d9:0x0] error -5.
2[4469]: segfault at 0 ip 0000000000403cf0 sp 00007ffe3b18ea28 error 6 in 2[400000+6000]
Lustre: 10852:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/19 is open, migrate only dentry
Lustre: 10852:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 19 previous similar messages
Lustre: dir [0x280000403:0x4ca:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 3513:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x240000403:0x9d9:0x0] : rc = -5
LustreError: 3513:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x9d9:0x0] error -5.
LustreError: 10835:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x658:0x0]: rc = -2
LustreError: 10835:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
Lustre: 17027:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xd31:0x0] with magic=0xbd60bd0
Lustre: 17027:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 18197:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x658:0x0]: rc = -2
LustreError: 18197:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 3744:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x240000404:0x658:0x0] mdc close failed: rc = -2
LustreError: 3744:0:(file.c:247:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Lustre: 12470:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 12470:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 65 previous similar messages
Lustre: dir [0x240000404:0x88a:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 26705:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0001: '17' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 17' to finish migration: rc = -1
LustreError: 26705:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/17 failed: rc = -1
LustreError: 26705:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 7 previous similar messages
LustreError: 23015:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 15' to finish migration: rc = -1
LustreError: 11216:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x749:0x0]: rc = -5
LustreError: 11216:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 8 previous similar messages
LustreError: 11216:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 11216:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 8 previous similar messages
LustreError: 17166:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0xaa7:0x0] migrate mdt count mismatch 3 != 1
LustreError: 18324:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x709:0x0]: rc = -2
LustreError: 18324:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
LustreError: 12666:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x240000404:0x709:0x0] mdc close failed: rc = -2
LustreError: 12666:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 30595:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0xbad:0x0] with magic=0xbd60bd0
Lustre: 30595:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 10848:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: '17' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 17' to finish migration: rc = -1
LustreError: 10848:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 1 previous similar message
Lustre: dir [0x240000403:0xb7d:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 8 previous similar messages
8[17381]: segfault at 8 ip 00007f5a325187e8 sp 00007ffd1f4f98d0 error 4 in ld-2.17.so[7f5a3250d000+22000]
12[19151]: segfault at 8 ip 00007f5fa5c0c7e8 sp 00007ffda5b27e30 error 4 in ld-2.17.so[7f5fa5c01000+22000]
LustreError: 14204:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x280000403:0x749:0x0] : rc = -5
LustreError: 14204:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x280000403:0x749:0x0] error -5.
LustreError: 10852:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=3 index=2 hash=crush:0x82000003 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 19244:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000404:0xaa2:0x0] error -5.
LustreError: 18063:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x1284:0x0] migrate mdt count mismatch 3 != 2
LustreError: 10848:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0001: '6' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 6' to finish migration: rc = -1
LustreError: 10848:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
LustreError: 20664:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0xc2c:0x0]/10 failed: rc = -2
LustreError: 20664:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 20 previous similar messages
LustreError: 18063:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x12c9:0x0] doesn't exist!: rc = -14
LustreError: 25413:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x11e2:0x0]: rc = -5
LustreError: 25413:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 29 previous similar messages
LustreError: 25413:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 25413:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 29 previous similar messages
LustreError: 17184:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x9b8:0x0] migrate mdt count mismatch 2 != 1
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 512 < left 788, rollback = 2
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 1502 previous similar messages
LustreError: 26244:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8800a8378958: inode [0x200000404:0x13b0:0x0] mdc close failed: rc = -13
LustreError: 26244:0:(file.c:247:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 18058:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1253:0x0] with magic=0xbd60bd0
Lustre: 18058:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 27 previous similar messages
2[28267]: segfault at 0 ip (null) sp 00007fffa9b7df68 error 14 in 2[400000+6000]
16[29930]: segfault at 8 ip 00007f091ae1a7e8 sp 00007ffdd6aca560 error 4 in ld-2.17.so[7f091ae0f000+22000]
LustreError: 18324:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xac3:0x0]: rc = -2
LustreError: 18324:0:(mdd_object.c:3901:mdd_close()) Skipped 1 previous similar message
19[32093]: segfault at 8 ip 00007f63fd5f77e8 sp 00007ffe3e7cafb0 error 4 in ld-2.17.so[7f63fd5ec000+22000]
Lustre: 22676:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0000: [0x200000403:0x1:0x0]/7 is open, migrate only dentry
Lustre: 22676:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 35 previous similar messages
2[32488]: segfault at 0 ip (null) sp 00007ffd7d7ba378 error 14 in 2[400000+6000]
LustreError: 31577:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8800a8378958: cannot apply new layout on [0x240000404:0xaa2:0x0] : rc = -5
LustreError: 31577:0:(lov_object.c:1341:lov_layout_change()) Skipped 1 previous similar message
LustreError: 26705:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0002: '12' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 12' to finish migration: rc = -1
LustreError: 26705:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
0[11096]: segfault at 8 ip 00007ff34cb847e8 sp 00007ffe71c38eb0 error 4 in ld-2.17.so[7ff34cb79000+22000]
2[13486]: segfault at 8 ip 00007fa228afe7e8 sp 00007ffd797b03b0 error 4 in ld-2.17.so[7fa228af3000+22000]
15[15943]: segfault at 0 ip 0000000000403e5f sp 00007fff31f55e40 error 6 in 15[400000+6000]
8[19828]: segfault at 8 ip 00007ff3c8c3d7e8 sp 00007ffc59da90b0 error 4 in ld-2.17.so[7ff3c8c32000+22000]
traps: 14[17426] general protection ip:40576f sp:7fff28b4f0e0 error:0 in 14[400000+6000]
Lustre: 12470:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12470:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 2564 previous similar messages
Lustre: 12470:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12470:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 2564 previous similar messages
Lustre: 12470:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0
Lustre: 12470:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 2564 previous similar messages
Lustre: 12470:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12470:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 2564 previous similar messages
Lustre: 12470:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12470:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 2564 previous similar messages
traps: 15[2561] general protection ip:40582f sp:7ffe5f2f8a28 error:0 in 15[400000+6000]
traps: 6[8675] trap invalid opcode ip:40537c sp:7ffd400af898 error:0 in 6[400000+6000]
Lustre: 6914:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7
Lustre: 6914:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 149 previous similar messages
4[15568]: segfault at 8 ip 00007f16803d17e8 sp 00007ffd65e27f50 error 4 in ld-2.17.so[7f16803c6000+22000]
15[15720]: segfault at 8 ip 00007fca73e637e8 sp 00007fff42e334b0 error 4 in ld-2.17.so[7fca73e58000+22000]
18[16869]: segfault at 0 ip 0000000000403cf0 sp 00007fffbd47a888 error 6 in 18[400000+6000]
10[23126]: segfault at 8 ip 00007f0977b777e8 sp 00007fff26baf810 error 4 in ld-2.17.so[7f0977b6c000+22000]
18[24745]: segfault at 8 ip 00007faa40e497e8 sp 00007ffe0f290be0 error 4 in ld-2.17.so[7faa40e3e000+22000]
5[24897]: segfault at 0 ip (null) sp 00007fffd7ca9158 error 14 in 5[400000+6000]
8[29762]: segfault at 8 ip 00007f16116457e8 sp 00007ffebd4da8b0 error 4 in ld-2.17.so[7f161163a000+22000]
LustreError: 23190:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e739ca88: inode [0x240000404:0x12b1:0x0] mdc close failed: rc = -13
LustreError: 23190:0:(file.c:247:ll_close_inode_openhandle()) Skipped 6 previous similar messages
Lustre: mdt00_016: service thread pid 18090 was inactive for 40.113 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 18090, comm: mdt00_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_reint_setattr+0x1324/0x15f0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 17027, comm: mdt00_029 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1729/0x2cc0 [mdt]
[<0>] mdt_intent_getattr+0x2cc/0x4e0 [mdt]
[<0>] mdt_intent_opc.constprop.74+0x211/0xc60 [mdt]
[<0>] mdt_intent_policy+0x10f/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x397/0x980 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x547/0x1900 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0000-osc-ffff8802e739ca88: disconnect after 20s idle
Lustre: mdt_io00_011: service thread pid 22676 was inactive for 72.206 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 22676, comm: mdt_io00_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x943/0xd80 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x2ea/0x810 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1b3/0x470 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0x57/0xf0 [mdt]
[<0>] mdt_reint_migrate+0x1832/0x24b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x73c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io00_000: service thread pid 10848 was inactive for 72.173 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_036: service thread pid 18546 was inactive for 72.066 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_022: service thread pid 32027 was inactive for 72.166 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
Lustre: mdt_io00_005: service thread pid 20664 was inactive for 72.234 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 13 previous similar messages
Lustre: lustre-OST0001-osc-ffff8800a8378958: disconnect after 22s idle
LustreError: 10629:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802cc115e00/0x89c6027d5ed9f30f lrc: 3/0,0 mode: PW/PW res: [0x280000404:0x161b:0x0].0x0 bits 0x4/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x89c6027d5ed9f301 expref: 752 pid: 16990 timeout: 930 lvb_type: 0
Lustre: 22676:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0002: [0x280000403:0x1:0x0]/17 is open, migrate only dentry
Lustre: 22676:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 6 previous similar messages
LustreError: 17018:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800a794d3d8 ns: mdt-lustre-MDT0002_UUID lock: ffff8802cea46d00/0x89c6027d5edc4b09 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 23 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x89c6027d5edc4ad8 expref: 232 pid: 17018 timeout: 0 lvb_type: 0
LustreError: 22676:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802c322cb40 x1829451678399360/t0(0) o105->lustre-MDT0002@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295
Lustre: mdt00_031: service thread pid 20979 completed after 89.657s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_040: service thread pid 29873 completed after 89.751s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_038: service thread pid 18563 completed after 90.106s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_012: service thread pid 18058 completed after 89.644s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 10830 completed after 90.144s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_039: service thread pid 29862 completed after 89.671s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_025: service thread pid 9562 completed after 90.126s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
Lustre: mdt00_022: service thread pid 32027 completed after 90.123s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 15348 completed after 89.668s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 10831 completed after 90.128s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_028: service thread pid 17018 completed after 89.630s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 13319 completed after 89.694s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 17166 completed after 89.718s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 6984:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -5
Lustre: mdt00_023: service thread pid 32068 completed after 90.148s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_029: service thread pid 17027 completed after 100.347s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_011: service thread pid 22676 completed after 100.433s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 18055 completed after 90.225s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 10053:0:(mdc_request.c:1464:mdc_read_page()) lustre-MDT0002-mdc-ffff8800a8378958: [0x280000403:0x1:0x0] lock enqueue fails: rc = -5
LustreError: 10016:0:(statahead.c:1801:is_first_dirent()) lustre: reading dir [0x280000403:0x1:0x0] at 0 stat_pid = 6105 : rc = -5
LustreError: 10053:0:(mdc_request.c:1464:mdc_read_page()) Skipped 3 previous similar messages
Lustre: mdt00_016: service thread pid 18090 completed after 100.567s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: dir [0x240000403:0x16de:0x0] stripe 1 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 17406:0:(ldlm_resource.c:982:ldlm_resource_complain()) lustre-MDT0002-mdc-ffff8800a8378958: namespace resource [0x280000403:0x1:0x0].0x0 (ffff88009b0f2340) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection restored to 192.168.123.22@tcp (at 0@lo)
18[18234]: segfault at 1c ip 00000000004017b0 sp 00007ffdd3d9fa80 error 6 in 18[400000+6000]
LustreError: 10629:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880098c87840/0x89c6027d5edbfb2a lrc: 3/0,0 mode: PR/PR res: [0x200000404:0x243c:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x89c6027d5edbfb0e expref: 1199 pid: 17184 timeout: 940 lvb_type: 0
Lustre: mdt00_021: service thread pid 2469 completed after 100.282s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_036: service thread pid 18546 completed after 100.089s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 18066 completed after 98.657s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_000: service thread pid 10848 completed after 100.730s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a8378958: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800a8378958: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 17363:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744701844 with bad export cookie 9927625166062191770
Lustre: mdt_io00_013: service thread pid 22763 completed after 99.892s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_005: service thread pid 20664 completed after 97.238s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8800a8378958: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt_io00_001: service thread pid 10851 completed after 97.264s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 16036:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
Lustre: mdt_io00_014: service thread pid 23015 completed after 97.204s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 16036:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 62 previous similar messages
Lustre: mdt_io00_007: service thread pid 19721 completed after 97.258s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_010: service thread pid 22669 completed after 97.316s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_017: service thread pid 26705 completed after 97.317s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 7295:0:(mdc_request.c:1464:mdc_read_page()) lustre-MDT0000-mdc-ffff8800a8378958: [0x200000403:0x1:0x0] lock enqueue fails: rc = -108
LustreError: 7295:0:(mdc_request.c:1464:mdc_read_page()) Skipped 12 previous similar messages
Lustre: mdt_io00_003: service thread pid 18501 completed after 96.977s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8800a8378958: Connection restored to 192.168.123.22@tcp (at 0@lo)
Lustre: mdt_io00_009: service thread pid 20961 completed after 96.884s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 1339, rollback = 2
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 641 previous similar messages
hrtimer: interrupt took 61122440 ns
LustreError: 19946:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000404:0x1688:0x0]/10 failed: rc = -2
LustreError: 19946:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 13 previous similar messages
1[20565]: segfault at 8 ip 00007f8d4be3a7e8 sp 00007fff009f97c0 error 4 in ld-2.17.so[7f8d4be2f000+22000]
LustreError: 10452:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0000: '5' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 5' to finish migration: rc = -1
LustreError: 10452:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 1 previous similar message
6[25620]: segfault at 8 ip 00007f43ae3557e8 sp 00007ffdae09d350 error 4 in ld-2.17.so[7f43ae34a000+22000]
LustreError: 17147:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000405:0x8f:0x0]: rc = -2
LustreError: 17147:0:(mdd_object.c:3901:mdd_close()) Skipped 3 previous similar messages
7[27876]: segfault at 8 ip 00007fbb023457e8 sp 00007ffc19835870 error 4 in ld-2.17.so[7fbb0233a000+22000]
5[31842]: segfault at 8 ip 00007f1a659357e8 sp 00007fffb71fa250 error 4 in ld-2.17.so[7f1a6592a000+22000]
2[3098]: segfault at 8 ip 00007fd0584457e8 sp 00007ffc16623b70 error 4 in ld-2.17.so[7fd05843a000+22000]
LustreError: 3481:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0x236:0x0]: rc = -5
LustreError: 3481:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 31 previous similar messages
LustreError: 3481:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 3481:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 31 previous similar messages
6[7005]: segfault at 0 ip 0000000000403cf0 sp 00007ffcfcc463e8 error 6 in 6[400000+6000]
Lustre: 18058:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x1998:0x0] with magic=0xbd60bd0
Lustre: 18058:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 125 previous similar messages
7[7128]: segfault at 0 ip (null) sp 00007ffe9b802178 error 14 in 7[400000+6000]
12[7565]: segfault at 0 ip 0000000000401d60 sp 00007ffe6dbd0b88 error 6 in 12[400000+6000]
16[10365]: segfault at 10 ip 00007fd95d685b43 sp 00007ffe196b7aa0 error 4 in ld-2.17.so[7fd95d67a000+22000]
Lustre: 20961:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0001: [0x240000403:0x1:0x0]/13 is open, migrate only dentry
19[15087]: segfault at 0 ip (null) sp 00007ffea8e97ca8 error 14 in 5[400000+6000]
LustreError: 10852:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 16990:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x3d3:0x0] migrate mdt count mismatch 3 != 1
LustreError: 30770:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x1800:0x0]: rc = -2
LustreError: 30770:0:(mdd_object.c:3901:mdd_close()) Skipped 3 previous similar messages
LustreError: 18101:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e739ca88: inode [0x240000404:0x1800:0x0] mdc close failed: rc = -2
LustreError: 18101:0:(file.c:247:ll_close_inode_openhandle()) Skipped 46 previous similar messages
2[22232]: segfault at 8 ip 00007f819da217e8 sp 00007ffc99f13f40 error 4 in ld-2.17.so[7f819da16000+22000]
LustreError: 30611:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x466:0x0] migrate mdt count mismatch 1 != 3
17[26923]: segfault at 0 ip 0000000000403e5f sp 00007ffe2f2588b0 error 6 in 17[400000+6000]
Lustre: dir [0x200000404:0x2bd5:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 22 previous similar messages
LustreError: 10629:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802be9e3880/0x89c6027d5eea8015 lrc: 3/0,0 mode: PW/PW res: [0x280000405:0x20f:0x0].0x0 bits 0x4/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x89c6027d5eea7ff2 expref: 278 pid: 18099 timeout: 1090 lvb_type: 0
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: operation mds_getxattr to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 31798:0:(mdc_request.c:1464:mdc_read_page()) lustre-MDT0002-mdc-ffff8800a8378958: [0x280000401:0x62:0x0] lock enqueue fails: rc = -108
LustreError: 31798:0:(mdc_request.c:1464:mdc_read_page()) Skipped 1 previous similar message
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection restored to 192.168.123.22@tcp (at 0@lo)
LustreError: 30098:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x280000405:0x496:0x0] : rc = -5
LustreError: 30098:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x280000405:0x496:0x0] error -5.
LustreError: 10852:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0002: '8' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 8' to finish migration: rc = -1
LustreError: 10852:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 10 previous similar messages
18[770]: segfault at 8 ip 00007f44c2b877e8 sp 00007ffd2892b8e0 error 4 in ld-2.17.so[7f44c2b7c000+22000]
17[882]: segfault at 8 ip 00007f4024f0a7e8 sp 00007fffe2bc24f0 error 4 in ld-2.17.so[7f4024eff000+22000]
LustreError: 17903:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1892:0x0] migrate mdt count mismatch 3 != 1
LustreError: 419:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 20 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 2314:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x200000405:0xa84:0x0] error -5.
15[5949]: segfault at 8 ip 00007f66f2e1e7e8 sp 00007ffe8387f7f0 error 4 in ld-2.17.so[7f66f2e13000+22000]
14[7580]: segfault at 8 ip 00007f572719e7e8 sp 00007ffeb42b2570 error 4 in ld-2.17.so[7f5727193000+22000]
5[7987]: segfault at 8 ip 00007f0f8ffca7e8 sp 00007ffe8a1203f0 error 4 in ld-2.17.so[7f0f8ffbf000+22000]
LustreError: 9499:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x240000403:0x1f2a:0x0] : rc = -5
LustreError: 9499:0:(lov_object.c:1341:lov_layout_change()) Skipped 2 previous similar messages
9[12332]: segfault at 8 ip 00007f2c3f6237e8 sp 00007ffd05f68e10 error 4 in ld-2.17.so[7f2c3f618000+22000]
12[12346]: segfault at 8 ip 00007fe2356687e8 sp 00007ffe06608a50 error 4 in ld-2.17.so[7fe23565d000+22000]
LustreError: 18005:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x1f88:0x0] migrate mdt count mismatch 1 != 3
LustreError: 18005:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
5[15197]: segfault at 8 ip 00007fefb66cb7e8 sp 00007ffd00880e50 error 4 in ld-2.17.so[7fefb66c0000+22000]
14[17493]: segfault at 8 ip 00007f5876c037e8 sp 00007ffd1d034790 error 4 in ld-2.17.so[7f5876bf8000+22000]
10[18013]: segfault at 8 ip 00007f88c24657e8 sp 00007ffcef0f8750 error 4 in ld-2.17.so[7f88c245a000+22000]
LustreError: 10851:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=1 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
Lustre: 18501:0:(mdt_reint.c:2460:mdt_reint_migrate()) lustre-MDT0002: [0x280000403:0x1:0x0]/0 is open, migrate only dentry
5[19605]: segfault at 8 ip 00007f649e1187e8 sp 00007ffc7855e8b0 error 4 in ld-2.17.so[7f649e10d000+22000]
Lustre: 6914:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6914:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3628 previous similar messages
Lustre: 6914:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6914:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3628 previous similar messages
Lustre: 6914:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0
Lustre: 6914:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3628 previous similar messages
Lustre: 6914:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6914:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3628 previous similar messages
Lustre: 6914:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6914:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3628 previous similar messages
4[24909]: segfault at 8 ip 00007f143fc337e8 sp 00007fff1ca54cf0 error 4 in ld-2.17.so[7f143fc28000+22000]
LustreError: 20750:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x221f:0x0] error -5.
LustreError: 20750:0:(vvp_io.c:1903:vvp_io_init()) Skipped 1 previous similar message
8[26850]: segfault at 8 ip 00007efc9d57e7e8 sp 00007ffd5f8d2230 error 4 in ld-2.17.so[7efc9d573000+22000]
LustreError: 18378:0:(mdd_object.c:3901:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x217c:0x0]: rc = -2
LustreError: 18378:0:(mdd_object.c:3901:mdd_close()) Skipped 10 previous similar messages
14[29241]: segfault at 8 ip 00007fa5d43057e8 sp 00007ffd29897460 error 4 in ld-2.17.so[7fa5d42fa000+22000]
LustreError: 20664:0:(mdt_reint.c:2540:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000404:0x3327:0x0]/13 failed: rc = -2
LustreError: 20664:0:(mdt_reint.c:2540:mdt_reint_migrate()) Skipped 77 previous similar messages
7[31289]: segfault at 8 ip 00007f877a1657e8 sp 00007fff3f3d58d0 error 4 in ld-2.17.so[7f877a15a000+22000]
LustreError: 131:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 2 [0x280000404:0x20f3:0x0] inode@0000000000000000: rc = -5
14[32089]: segfault at 8 ip 00007f03b4af67e8 sp 00007ffd5e274050 error 4 in ld-2.17.so[7f03b4aeb000+22000]
14[32071]: segfault at 8 ip 00007fd8eeecb7e8 sp 00007ffc74c18010 error 4 in ld-2.17.so[7fd8eeec0000+22000]
LustreError: 29268:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x200000405:0x1001:0x0] : rc = -5
LustreError: 29268:0:(lov_object.c:1341:lov_layout_change()) Skipped 1 previous similar message
LustreError: 9562:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0000: [0x200000405:0xfbe:0x0] migrate mdt count mismatch 3 != 1
LustreError: 9562:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 1 previous similar message
16[1840]: segfault at 8 ip 00007f12d244a7e8 sp 00007ffd7dfbced0 error 4 in ld-2.17.so[7f12d243f000+22000]
LustreError: 10851:0:(lustre_lmv.h:500:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
4[1372]: segfault at 8 ip 00007f245075e7e8 sp 00007ffded4fa070 error 4 in ld-2.17.so[7f2450753000+22000]
1[11399]: segfault at 1c ip 00000000004017b0 sp 00007ffe337a2a20 error 6 in 1[400000+6000]
14[8578]: segfault at 0 ip (null) sp 00007ffebfbf16c8 error 14 in 14 (deleted)[400000+6000]
19[12552]: segfault at 8 ip 00007fcf97e967e8 sp 00007ffdbb8ff5a0 error 4 in ld-2.17.so[7fcf97e8b000+22000]
LustreError: 13991:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0x1092:0x0]: rc = -5
LustreError: 13991:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 115 previous similar messages
LustreError: 13991:0:(llite_lib.c:3696:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 13991:0:(llite_lib.c:3696:ll_prep_inode()) Skipped 115 previous similar messages
Lustre: 21316:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7
Lustre: 21316:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 343 previous similar messages
15[21880]: segfault at 8 ip 00007fb64bd337e8 sp 00007fff9f9b12e0 error 4 in ld-2.17.so[7fb64bd28000+22000]
16[23082]: segfault at 8 ip 00007fee14d8b7e8 sp 00007fff6334b120 error 4 in ld-2.17.so[7fee14d80000+22000]
6[24462]: segfault at 8 ip 00007f8c08e277e8 sp 00007ffc2c50e500 error 4 in ld-2.17.so[7f8c08e1c000+22000]
10[24874]: segfault at 1c ip 00000000004017b0 sp 00007ffefac31540 error 6 in 10[400000+6000]
1[398]: segfault at 8 ip 00007f6f45cc57e8 sp 00007fff6ad53760 error 4 in ld-2.17.so[7f6f45cba000+22000]
10[32061]: segfault at 8 ip 00007f2aa23f27e8 sp 00007ffcb70c9840 error 4 in ld-2.17.so[7f2aa23e7000+22000]
13[596]: segfault at 8 ip 00007f715f4997e8 sp 00007fff5f5f9be0 error 4 in ld-2.17.so[7f715f48e000+22000]
15[3407]: segfault at 8 ip 00007fbc16cf77e8 sp 00007fff9f717ec0 error 4 in ld-2.17.so[7fbc16cec000+22000]
Lustre: 31070:0:(lod_lov.c:1417:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0x1619:0x0] with magic=0xbd60bd0
Lustre: 31070:0:(lod_lov.c:1417:lod_parse_striping()) Skipped 101 previous similar messages
16[7658]: segfault at 1c ip 00000000004017b0 sp 00007ffcba58bf90 error 6 in 16[400000+6000]
Lustre: dir [0x280000406:0x9dc:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 22 previous similar messages
1[12768]: segfault at 8 ip 00007f443a5a97e8 sp 00007ffd151a4450 error 4 in ld-2.17.so[7f443a59e000+22000]
LustreError: 31145:0:(mdt_open.c:1302:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x39fb:0x0] doesn't exist!: rc = -14
0[14536]: segfault at 0 ip 0000000000403e5f sp 00007ffc5aa09300 error 6 in 0[400000+6000]
LustreError: 14552:0:(lov_object.c:1341:lov_layout_change()) lustre-clilov-ffff8802e739ca88: cannot apply new layout on [0x240000403:0x2974:0x0] : rc = -5
LustreError: 14552:0:(lov_object.c:1341:lov_layout_change()) Skipped 8 previous similar messages
LustreError: 14552:0:(vvp_io.c:1903:vvp_io_init()) lustre: refresh file layout [0x240000403:0x2974:0x0] error -5.
LustreError: 14552:0:(vvp_io.c:1903:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 31109:0:(mdt_xattr.c:402:mdt_dir_layout_update()) lustre-MDT0002: [0x280000406:0xbd0:0x0] migrate mdt count mismatch 2 != 1
LustreError: 31109:0:(mdt_xattr.c:402:mdt_dir_layout_update()) Skipped 2 previous similar messages
LustreError: 20664:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) lustre-MDD0002: '14' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 14' to finish migration: rc = -1
LustreError: 20664:0:(mdd_dir.c:4662:mdd_migrate_cmd_check()) Skipped 18 previous similar messages
9[20130]: segfault at 8 ip 00007f04f49f47e8 sp 00007ffc62a725a0 error 4 in ld-2.17.so[7f04f49e9000+22000]
5[25773]: segfault at 8 ip 00007f0fca7177e8 sp 00007fff6458d620 error 4 in ld-2.17.so[7f0fca70c000+22000]
0[26183]: segfault at 8 ip 00007f261f4087e8 sp 00007ffc61d24ba0 error 4 in ld-2.17.so[7f261f3fd000+22000]
5[31465]: segfault at 8 ip 00007f5a67ae47e8 sp 00007ffdeb92dda0 error 4 in ld-2.17.so[7f5a67ad9000+22000]
dd (428) used greatest stack depth: 10016 bytes left
9[2171]: segfault at 8 ip 00007f2e277137e8 sp 00007fff903fa7f0 error 4 in ld-2.17.so[7f2e27708000+22000]
Lustre: 26631:0:(mdd_dir.c:4741:mdd_migrate_object()) lustre-MDD0001: [0x240000403:0x1:0x0]/14 is open, migrate only dentry
Lustre: 26631:0:(mdd_dir.c:4741:mdd_migrate_object()) Skipped 175 previous similar messages
LustreError: 10629:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff880098ebd680/0x89c6027d5f563562 lrc: 3/0,0 mode: PR/PR res: [0x280000406:0xe67:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x89c6027d5f563546 expref: 732 pid: 12909 timeout: 1543 lvb_type: 0
LustreError: 32101:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802dcc0dd28 ns: mdt-lustre-MDT0002_UUID lock: ffff8802a8a8f480/0x89c6027d5f566519 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x89c6027d5f56650b expref: 217 pid: 32101 timeout: 0 lvb_type: 0
LustreError: 32101:0:(ldlm_lockd.c:1447:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: 19078:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744702447 with bad export cookie 9927625166073124937
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0002-mdc-ffff8800a8378958: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 19078:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message
LustreError: 4892:0:(file.c:6136:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -5
LustreError: 4892:0:(file.c:6136:ll_inode_revalidate_fini()) Skipped 68 previous similar messages
LustreError: 4405:0:(mdc_request.c:1464:mdc_read_page()) lustre-MDT0002-mdc-ffff8800a8378958: [0x280000401:0x84:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0002-mdc-ffff8800a8378958: Connection restored to 192.168.123.22@tcp (at 0@lo)
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 513 < left 2168, rollback = 2
Lustre: 19946:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 5533 previous similar messages
traps: 17[9297] trap invalid opcode ip:404850 sp:7fff85486fc8 error:0 in 17[400000+6000]
19[11069]: segfault at 8 ip 00007f404d2367e8 sp 00007ffd4e29b7f0 error 4 in ld-2.17.so[7f404d22b000+22000]
4[13827]: segfault at 406000 ip 0000000000406000 sp 00007ffce787b918 error 14 in 4[606000+1000]
3[21089]: segfault at 8 ip 00007fd759f5f7e8 sp 00007ffd20ef43e0 error 4 in ld-2.17.so[7fd759f54000+22000]