NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [mdt06_003:10763]
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk i2c_core serio_raw libata [last unloaded: libcfs]
CPU: 12 PID: 10763 Comm: mdt06_003 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88024ee18010 ti: ffff880266538000 task.ti: ffff880266538000
RIP: 0010:[<ffffffffa0cff778>] [<ffffffffa0cff778>] zap_leaf_lookup+0xc8/0x160 [zfs]
RSP: 0018:ffff88026653b400 EFLAGS: 00000287
RAX: 0000000000000000 RBX: ffff88026653b390 RCX: 0000000000000009
RDX: ffff8801213b9430 RSI: 0000000000000200 RDI: 75dc52f175650000
RBP: ffff88026653b430 R08: 0000000000000000 R09: ffff8801213b9000
R10: 0000000000000000 R11: 000000000000000c R12: ffff8802afaa3c88
R13: ffff88026653b468 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8c54df99e5 CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0cfd55c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c60acd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0cfd651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0d02dfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0d01a97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0d03085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa0b08e08>] osd_dir_insert+0x228/0x9b0 [osd_zfs]
[<ffffffffa13d52ad>] lod_sub_insert+0x1cd/0x3b0 [lod]
[<ffffffffa13bce8f>] ? lod_create+0x28f/0x3b0 [lod]
[<ffffffffa12391a1>] ? mdd_create_object_internal+0xb1/0x360 [mdd]
[<ffffffffa13a28b6>] lod_insert+0x16/0x20 [lod]
[<ffffffffa121b146>] __mdd_index_insert_only+0xc6/0x2d0 [mdd]
[<ffffffffa121df09>] __mdd_index_insert+0x49/0x2e0 [mdd]
[<ffffffffa1226d6d>] mdd_create+0x117d/0x1e30 [mdd]
[<ffffffffa12d2703>] mdt_reint_open+0x21c3/0x2e40 [mdt]
[<ffffffffa12ad5c3>] ? old_init_ucred_common+0x163/0x4c0 [mdt]
[<ffffffffa12c2b17>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa129783c>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<ffffffffa12a3bc3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa129b9e9>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<ffffffffa12a3b30>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa12a168a>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa065c501>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<ffffffffa019cdb5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa01a0026>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa06849bb>] ldlm_handle_enqueue+0x35b/0x1820 [ptlrpc]
[<ffffffffa0607300>] ? lustre_msg_buf_v2+0x1e0/0x1f0 [ptlrpc]
[<ffffffffa06ca138>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa06d497e>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<ffffffffa0619a3c>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<ffffffffa061b7e6>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<ffffffffa061ab70>] ? ptlrpc_wait_event+0x610/0x610 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 11910:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880251391940 x1796836157959872/t4295151898(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:107/0 lens 376/39032 e 0 to 0 dl 1713599447 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 11910:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 1 previous similar message
Lustre: 32157:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880286815f40 x1796836160002176/t4295119306(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:141/0 lens 376/35080 e 0 to 0 dl 1713599481 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 5081:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802072f1e40 x1796836159378432/t4295102558(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:147/0 lens 376/38656 e 0 to 0 dl 1713599487 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 25128:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802be2a8540 x1796836164451264/t4295119755(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:188/0 lens 376/39280 e 0 to 0 dl 1713599528 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 20666:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802553f7340 x1796836165569280/t4295103354(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:196/0 lens 376/38656 e 0 to 0 dl 1713599536 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 17102:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800030fee40 x1796836164922624/t4295119927(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:210/0 lens 376/39280 e 0 to 0 dl 1713599550 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 21122:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88024543da40 x1796836170641664/t4295104169(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:254/0 lens 376/38656 e 0 to 0 dl 1713599594 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 32162:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802f92ed540 x1796836175163200/t4295154453(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:290/0 lens 376/44704 e 0 to 0 dl 1713599630 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 32162:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 3 previous similar messages
Lustre: 20059:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88015a9abc40 x1796836180481216/t4295154831(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:354/0 lens 376/47368 e 0 to 0 dl 1713599694 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 20059:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 2 previous similar messages
LustreError: 1492:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 1492:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 1492:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 1492:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 1492:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 1492:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: 28903:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880165672840 x1796836196005504/t4295121976(0) o101->a345ccef-4441-49ee-abcf-adbbc26f73c1@0@lo:513/0 lens 376/47992 e 0 to 0 dl 1713599853 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 28903:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 4 previous similar messages
ptlrpc_watchdog_fire: 11 callbacks suppressed
Lustre: mdt_io06_002: service thread pid 1291 was inactive for 52.118 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 1291, comm: mdt_io06_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] mdt_reint_rename+0x28b0/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 17028:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88002715a340 x1796836226227968/t4295110216(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:87/0 lens 376/48512 e 0 to 0 dl 1713600182 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 17028:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 18 previous similar messages
Lustre: mdt_io03_006: service thread pid 25843 was inactive for 76.091 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 25843, comm: mdt_io03_006 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_check_lock+0xec/0x3c0 [mdt]
[<0>] mdt_reint_rename+0x277b/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 1112:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880096fd3c40/0x90236adea2f70189 lrc: 3/0,0 mode: PR/PR res: [0x200000406:0xcc3:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x90236adea2f7016d expref: 55 pid: 25245 timeout: 21260 lvb_type: 0
LustreError: 32106:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff8802d4306678 ns: mdt-lustre-MDT0000_UUID lock: ffff8802d6872980/0x90236adea30254d2 lrc: 3/0,0 mode: PR/PR res: [0x240000406:0x1e56:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x90236adea2f7ebc3 expref: 3 pid: 32106 timeout: 0 lvb_type: 0
LustreError: 32106:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) Skipped 14 previous similar messages
LustreError: 11-0: lustre-MDT0000-mdc-ffff880214fb92a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff880214fb92a8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff880214fb92a8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 25738:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880214fb92a8: inode [0x200000403:0x2:0x0] mdc close failed: rc = -108
LustreError: 25738:0:(file.c:264:ll_close_inode_openhandle()) Skipped 34 previous similar messages
LustreError: 25738:0:(ldlm_resource.c:1128:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff880214fb92a8: namespace resource [0x200000406:0xcc3:0x0].0x0 (ffff8800794ab4c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 25738:0:(ldlm_resource.c:1128:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff880214fb92a8: Connection restored to (at 0@lo)
Lustre: mdt_io07_003: service thread pid 31868 was inactive for 40.042 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 31868, comm: mdt_io07_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_check_lock+0xec/0x3c0 [mdt]
[<0>] mdt_reint_rename+0x277b/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0003-osc-MDT0002: update sequence from 0x380000400 to 0x380000403
Lustre: lustre-OST0000-osc-MDT0002: update sequence from 0x2c0000400 to 0x2c0000403
Lustre: mdt_io05_002: service thread pid 1288 was inactive for 40.116 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 1288, comm: mdt_io05_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x4d9/0x7e0 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_lock_two_dirs+0x31/0x1b0 [mdt]
[<0>] mdt_reint_rename+0x12b1/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0001-osc-MDT0002: update sequence from 0x300000400 to 0x300000403
Lustre: 20846:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009f5dc640 x1796836276818176/t4295160236(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:532/0 lens 376/48328 e 0 to 0 dl 1713600627 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 20846:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 18 previous similar messages
Lustre: lustre-OST0002-osc-MDT0002: update sequence from 0x340000400 to 0x340000403
Lustre: ll_ost03_022: service thread pid 28167 was inactive for 40.037 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 28167, comm: ll_ost03_022 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] cv_wait_common+0xb0/0x150 [spl]
[<0>] __cv_wait_io+0x18/0x20 [spl]
[<0>] txg_wait_synced_impl+0xdd/0x130 [zfs]
[<0>] txg_wait_synced+0x10/0x50 [zfs]
[<0>] dmu_tx_wait+0x26d/0x3a0 [zfs]
[<0>] dmu_tx_assign+0x91/0x480 [zfs]
[<0>] osd_trans_start+0x1a7/0x480 [osd_zfs]
[<0>] ofd_precreate_objects+0xa1a/0x2190 [ofd]
[<0>] ofd_create_hdl+0x421/0x1d30 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0003-osc-MDT0001: update sequence from 0x380000401 to 0x380000404
Lustre: lustre-OST0001-osc-MDT0001: update sequence from 0x300000401 to 0x300000404
Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x2c0000401 to 0x2c0000404
Lustre: lustre-OST0002-osc-MDT0001: update sequence from 0x340000401 to 0x340000404
Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000405
Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x300000402 to 0x300000405
Lustre: mdt_io01_001: service thread pid 1275 was inactive for 42.128 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 1275, comm: mdt_io01_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_rename_source_lock+0xa9/0xd6 [mdt]
[<0>] mdt_reint_rename+0x261a/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x380000402 to 0x380000405
Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x340000402 to 0x340000405
Lustre: mdt_io03_002: service thread pid 1282 was inactive for 40.106 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 1282, comm: mdt_io03_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_check_lock+0xec/0x3c0 [mdt]
[<0>] mdt_reint_rename+0x277b/0x2af0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 17049:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800aee44140 x1796836328993216/t4295164026(0) o101->9b955bb2-a6c1-435a-be3c-b467bee98c35@0@lo:410/0 lens 376/48328 e 0 to 0 dl 1713601260 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: 17049:0:(mdt_recovery.c:148:mdt_req_from_lrd()) Skipped 38 previous similar messages
LustreError: 20108:0:(out_lib.c:1188:out_tx_index_delete_undo()) lustre-MDT0001-osd: Oops, can not rollback index_delete yet: rc = -524
LustreError: 20632:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x57b7:0x0] doesn't exist!: rc = -14
LustreError: 1492:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 1492:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 1492:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 1492:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 25097:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x57b7:0x0] doesn't exist!: rc = -14
LustreError: 25097:0:(mdt_open.c:1280:mdt_cross_open()) Skipped 1 previous similar message
LustreError: 5657:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x57b7:0x0] doesn't exist!: rc = -14
LustreError: 10575:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x57b7:0x0] doesn't exist!: rc = -14
LustreError: 29516:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0000: [0x200000404:0x57b7:0x0] doesn't exist!: rc = -14
LustreError: 29516:0:(mdt_open.c:1280:mdt_cross_open()) Skipped 3 previous similar messages

NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [mdt04_020:20258]
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy i2c_core serio_raw virtio_blk libata [last unloaded: libcfs]
CPU: 8 PID: 20258 Comm: mdt04_020 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88021ee5dc40 ti: ffff8802c37b4000 task.ti: ffff8802c37b4000
RIP: 0010:[<ffffffffa0d0a778>] [<ffffffffa0d0a778>] zap_leaf_lookup+0xc8/0x160 [zfs]
RSP: 0018:ffff8802c37b7400 EFLAGS: 00000207
RAX: 0000000000000000 RBX: ffff8802c37b7390 RCX: 0000000000000009
RDX: ffff8802662a1430 RSI: 0000000000000200 RDI: c712ee3542370000
RBP: ffff8802c37b7430 R08: 0000000000000000 R09: ffff8802662a1000
R10: 0000000000000000 R11: 0000000000000008 R12: 0000000000000000
R13: ffff8802c37b7468 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001b1894c CR3: 000000027a506000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0d0855c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c6bacd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0d08651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0d0ddfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0d0ca97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0d0e085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa0afae08>] osd_dir_insert+0x228/0x9b0 [osd_zfs]
[<ffffffffa13e02ad>] lod_sub_insert+0x1cd/0x3b0 [lod]
[<ffffffffa13c7e8f>] ? lod_create+0x28f/0x3b0 [lod]
[<ffffffffa12441a1>] ? mdd_create_object_internal+0xb1/0x360 [mdd]
[<ffffffffa13ad8b6>] lod_insert+0x16/0x20 [lod]
[<ffffffffa1226146>] __mdd_index_insert_only+0xc6/0x2d0 [mdd]
[<ffffffffa1228f09>] __mdd_index_insert+0x49/0x2e0 [mdd]
[<ffffffffa1231d6d>] mdd_create+0x117d/0x1e30 [mdd]
[<ffffffffa12dd703>] mdt_reint_open+0x21c3/0x2e40 [mdt]
[<ffffffffa12b85c3>] ? old_init_ucred_common+0x163/0x4c0 [mdt]
[<ffffffffa12cdb17>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa12a283c>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<ffffffffa12aebc3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa12a69e9>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<ffffffffa12aeb30>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa12ac68a>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa0656501>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<ffffffffa0199db5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa019d026>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa067e9bb>] ldlm_handle_enqueue+0x35b/0x1820 [ptlrpc]
[<ffffffffa0601300>] ? lustre_msg_buf_v2+0x1e0/0x1f0 [ptlrpc]
[<ffffffffa06c4138>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa06ce97e>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<ffffffffa0613a3c>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<ffffffffa06157e6>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffffa0614b70>] ? ptlrpc_wait_event+0x610/0x610 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

LustreError: 16133:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/19 failed: rc = -114
LustreError: 326:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88027dd04a88: inode [0x280000404:0xc:0x0] mdc close failed: rc = -116
Lustre: 16069:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880029aee940 x1796702372696768/t4294968345(0) o101->e6196351-bfe5-4337-9751-10be454bf25c@0@lo:300/0 lens 384/840 e 0 to 0 dl 1713469025 ref 1 fl Interpret:H/202/0 rc 0/0 job:'cat.0' uid:0 gid:0
LustreError: 16112:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/8 failed: rc = -114
LustreError: 815:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88027dd04a88: inode [0x240000403:0x4b:0x0] mdc close failed: rc = -116
LustreError: 16124:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/18 failed: rc = -114
LustreError: 1241:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x200000404:0x3c:0x0] mdc close failed: rc = -116
Lustre: 16055:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802bd0f0a40 x1796702372912000/t4294968533(0) o101->c04290d3-223d-4094-bf7d-54deef380064@0@lo:303/0 lens 384/864 e 0 to 0 dl 1713469028 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 16121:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/0 failed: rc = -114
LustreError: 1714:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88027dd04a88: inode [0x280000403:0x10:0x0] mdc close failed: rc = -116
LustreError: 16127:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x200000403:0x1:0x0]/12 failed: rc = -16
LustreError: 16127:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 3786:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x280000403:0x67:0x0] mdc close failed: rc = -116
LustreError: 3444:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '14' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 14' to finish migration: rc = -1
13[1606]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffc34be3d60 error 14
LustreError: 16132:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/18 failed: rc = -114
LustreError: 16132:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 8 previous similar messages
LustreError: 3015:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '12' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 12' to finish migration: rc = -1
LustreError: 16118:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '12' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 12' to finish migration: rc = -1
LustreError: 3015:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/0 failed: rc = -16
LustreError: 3015:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 10 previous similar messages
LustreError: 25918:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x280000403:0x5f:0x0] mdc close failed: rc = -2
LustreError: 25918:0:(file.c:264:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x240000404:0xff:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 16030:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x101:0x0]: rc = -5
LustreError: 16030:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 18019:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88027dd04a88: inode [0x200000404:0x3:0x0] mdc close failed: rc = -116
Lustre: dir [0x280000404:0xa3:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
Lustre: 26869:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0xb1:0x0] with magic=0xbd60bd0
LustreError: 3444:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000404:0x16b:0x0]/18 failed: rc = -114
LustreError: 3444:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 32 previous similar messages
Lustre: dir [0x200000404:0x14e:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 47 previous similar messages
Lustre: 2347:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802061d4b40 x1796702377123456/t4294971073(0) o101->c04290d3-223d-4094-bf7d-54deef380064@0@lo:413/0 lens 376/864 e 0 to 0 dl 1713469138 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 5371:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '19' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 19' to finish migration: rc = -1
LustreError: 25533:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x172:0x0]: rc = -5
LustreError: 25533:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 25533:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 25533:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: dir [0x280000404:0x16b:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 28445:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x172:0x0]: rc = -5
LustreError: 28445:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 28445:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 28445:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 27296:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x237:0x0] with magic=0xbd60bd0
Lustre: 27296:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
1[29359]: segfault at 8 ip 00007f925e5b57e8 sp 00007ffed7eb62a0 error 4 in ld-2.17.so[7f925e5aa000+22000]
LustreError: 2966:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '17' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 17' to finish migration: rc = -1
Lustre: 1408:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x2bd:0x0] with magic=0xbd60bd0
Lustre: 1408:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 31684:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x280000404:0x16b:0x0] mdc close failed: rc = -2
LustreError: 31684:0:(file.c:264:ll_close_inode_openhandle()) Skipped 8 previous similar messages
Lustre: 2347:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x199:0x0] with magic=0xbd60bd0
Lustre: 2347:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 15 previous similar messages
LustreError: 16123:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '11' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 11' to finish migration: rc = -1
Lustre: dir [0x240000404:0x2a0:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000403:0x18f:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 11258:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x349:0x0]: rc = -5
LustreError: 11258:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 11258:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 11258:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 27762:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x413:0x0] with magic=0xbd60bd0
Lustre: 27762:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 2795:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/2 failed: rc = -16
LustreError: 2795:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 75 previous similar messages
LustreError: 6647:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x229:0x0]: rc = -5
LustreError: 6647:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 6647:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 6647:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 13660:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x349:0x0]: rc = -5
LustreError: 13660:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 16113:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '3' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 3' to finish migration: rc = -1
LustreError: 16925:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x240000403:0x349:0x0] : rc = -5
LustreError: 16925:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x240000403:0x349:0x0] error -5.
LustreError: 22318:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x349:0x0]: rc = -5
LustreError: 22318:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 22318:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 22318:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 26510:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x240000403:0x349:0x0] : rc = -5
LustreError: 26510:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
Lustre: mdt_io02_002: service thread pid 16118 was inactive for 40.106 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16118, comm: mdt_io02_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x4d9/0x7e0 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_reint_migrate+0xe24/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt01_012: service thread pid 1713 was inactive for 40.155 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 1713, comm: mdt01_012 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1699/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1820 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io02_003: service thread pid 8045 was inactive for 40.064 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 8045, comm: mdt_io02_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x377/0x420 [mdt]
[<0>] mdt_rename_lock+0xc3/0x2d0 [mdt]
[<0>] mdt_reint_migrate+0x891/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io02_004: service thread pid 23426 was inactive for 40.032 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io06_003: service thread pid 6135 was inactive for 72.015 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io07_004: service thread pid 4301 was inactive for 72.249 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
Lustre: mdt_io03_004: service thread pid 2966 was inactive for 72.112 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
Lustre: mdt_io00_005: service thread pid 10266 was inactive for 74.137 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 15997:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88028a589e40/0x2e9add47afe47a27 lrc: 3/0,0 mode: CR/CR res: [0x240000403:0x43e:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x2e9add47afe47a0b expref: 231 pid: 30931 timeout: 23716 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff880270f05d28: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff880270f05d28: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff880270f05d28: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 21363:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x240000403:0x43e:0x0] mdc close failed: rc = -5
LustreError: 21363:0:(file.c:264:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: dir [0x240000403:0x337:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 14215:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff880270f05d28: [0x240000403:0x44d:0x0] lock enqueue fails: rc = -108
LustreError: 19942:0:(llite_lib.c:2019:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 27853:0:(ldlm_resource.c:1128:ldlm_resource_complain()) lustre-MDT0001-mdc-ffff880270f05d28: namespace resource [0x240000403:0x43e:0x0].0x0 (ffff880323326ac0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0001-mdc-ffff880270f05d28: Connection restored to (at 0@lo)
LustreError: 10266:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '10' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 10' to finish migration: rc = -1
LustreError: 29309:0:(osd_index.c:221:__osd_xattr_load_by_oid()) lustre-MDT0001: can't get bonus, rc = -2
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000403:0x337:0x0]/0xa): rc = 0
LustreError: 13298:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/4 failed: rc = -16
LustreError: 13298:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 31 previous similar messages
LustreError: 30278:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x240000403:0x349:0x0] : rc = -5
LustreError: 30278:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x349:0x0]: rc = -5
LustreError: 30278:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 30278:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 30278:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: 1841:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x399:0x0] with magic=0xbd60bd0
Lustre: 1841:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 11 previous similar messages
LustreError: 2166:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x240000403:0x349:0x0] : rc = -5
6[3238]: segfault at 8 ip 00007f55588027e8 sp 00007ffcd96076a0 error 4 in ld-2.17.so[7f55587f7000+22000]
Lustre: 28533:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x511:0x0] with magic=0xbd60bd0
Lustre: 28533:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 19 previous similar messages
Lustre: dir [0x280000403:0x32d:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 16074:0:(mdd_object.c:3873:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x32d:0x0]: rc = -2
16[22951]: segfault at 8 ip 00007f4ebdd3a7e8 sp 00007ffd55b8b6c0 error 4 in ld-2.17.so[7f4ebdd2f000+22000]
19[23460]: segfault at 8 ip 00007f848020d7e8 sp 00007fffa4f70ed0 error 4 in ld-2.17.so[7f8480202000+22000]
LustreError: 16130:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '2' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 2' to finish migration: rc = -1
LustreError: 16130:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
Lustre: 4466:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000405:0x160:0x0] with magic=0xbd60bd0
Lustre: 4466:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 19767:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x637:0x0]: rc = -5
LustreError: 19767:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 31 previous similar messages
LustreError: 19767:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 19767:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 31 previous similar messages
16[29328]: segfault at 8 ip 00007ff3df8517e8 sp 00007ffd597fd320 error 4 in ld-2.17.so[7ff3df846000+22000]
16[1897]: segfault at 8 ip 00007f2fc47bb7e8 sp 00007fff3dfda320 error 4 in ld-2.17.so[7f2fc47b0000+22000]
5[3000]: segfault at 8 ip 00007fa7a55577e8 sp 00007ffddb91ae20 error 4 in ld-2.17.so[7fa7a554c000+22000]
LustreError: 31233:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 18 [0x280000403:0x4ae:0x0]: rc = -5
8[5368]: segfault at 8 ip 00007ff915aa27e8 sp 00007fffc3bacb70 error 4 in ld-2.17.so[7ff915a97000+22000]
16[4925]: segfault at 8 ip 00007f8299fa47e8 sp 00007fff94423c80 error 4 in ld-2.17.so[7f8299f99000+22000]
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000404:0x577:0x0]/0xa): rc = 0
LustreError: 30607:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 18 [0x240000403:0x4f8:0x0]: rc = -5
9[20718]: segfault at 8 ip 00007f73b36f37e8 sp 00007fffa86ada40 error 4 in ld-2.17.so[7f73b36e8000+22000]
LustreError: 26036:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x280000404:0xa2c:0x0] mdc close failed: rc = -116
LustreError: 26036:0:(file.c:264:ll_close_inode_openhandle()) Skipped 29 previous similar messages
Lustre: dir [0x200000403:0x997:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
LustreError: 26857:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x6eb:0x0] migrate mdt count mismatch 3 != 2
LustreError: 20351:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0xb5a:0x0] ACL: rc = -2
5[10675]: segfault at 8 ip 00007faf6e5c47e8 sp 00007ffd91deef10 error 4 in ld-2.17.so[7faf6e5b9000+22000]
LustreError: 26415:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '1' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 1' to finish migration: rc = -1
LustreError: 26415:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 10 previous similar messages
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000404:0xbe2:0x0]/0xa): rc = 0
LustreError: 17306:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xbd9:0x0]: rc = -5
LustreError: 17306:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 17 previous similar messages
LustreError: 17306:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 17306:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 17 previous similar messages
16[22204]: segfault at 8 ip 00007eff6e78d7e8 sp 00007ffd614d60b0 error 4 in ld-2.17.so[7eff6e782000+22000]
Lustre: 27687:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xc07:0x0] with magic=0xbd60bd0
Lustre: 27687:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 9 previous similar messages
16[28518]: segfault at 8 ip 00007f4aa3adb7e8 sp 00007ffc3f195270 error 4 in ld-2.17.so[7f4aa3ad0000+22000]
0[2977]: segfault at 8 ip 00007f485e0447e8 sp 00007ffd8e6ce7c0 error 4 in ld-2.17.so[7f485e039000+22000]
LustreError: 8904:0:(llite_lib.c:1868:ll_update_lsm_md()) lustre: [0x280000403:0xb04:0x0] dir layout mismatch:
LustreError: 8904:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x3 count=2 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 8904:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) stripe[0] [0x280000400:0x4d:0x0]
LustreError: 8904:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=5 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:3 pool=
LustreError: 13173:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/19 failed: rc = -114
LustreError: 13173:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 250 previous similar messages
17[11643]: segfault at 0 ip (null) sp 00007fff4bf39a28 error 14 in 17[400000+6000]
9[19320]: segfault at 0 ip (null) sp 00007ffd43664d48 error 14 in 9[400000+6000]
Lustre: dir [0x200000404:0xe7d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 51 previous similar messages
LustreError: 20413:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 19 [0x200000403:0x127e:0x0]: rc = -5
LustreError: 6274:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x200000403:0x127e:0x0] : rc = -5
LustreError: 6274:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x200000403:0x127e:0x0] error -5.
LustreError: 19203:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 11 [0x240000403:0x1097:0x0]: rc = -5
LustreError: 19203:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 1 previous similar message
LustreError: 18937:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x200000403:0x127e:0x0] : rc = -5
LustreError: 13518:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 19 [0x200000403:0x127e:0x0]: rc = -5
8[20717]: segfault at 8 ip 00007f3405bb07e8 sp 00007ffe82a9b990 error 4 in ld-2.17.so[7f3405ba5000+22000]
Lustre: 1740:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x135f:0x0] with magic=0xbd60bd0
Lustre: 1740:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 31 previous similar messages
LustreError: 5672:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880270f05d28: inode [0x200000404:0x1495:0x0] mdc close failed: rc = -13
LustreError: 5672:0:(file.c:264:ll_close_inode_openhandle()) Skipped 24 previous similar messages
15[13379]: segfault at 8 ip 00007ffb5f2567e8 sp 00007ffece3daa30 error 4 in ld-2.17.so[7ffb5f24b000+22000]
LustreError: 12202:0:(llite_lib.c:1868:ll_update_lsm_md()) lustre: [0x200000404:0x15d7:0x0] dir layout mismatch:
LustreError: 12202:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 12202:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) stripe[0] [0x200000400:0x61:0x0]
LustreError: 12202:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) Skipped 6 previous similar messages
LustreError: 12202:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:3 pool=
2[24193]: segfault at 8 ip 00007f4f40e2f7e8 sp 00007ffe2e647c10 error 4 in ld-2.17.so[7f4f40e24000+22000]
LustreError: 27868:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x200000403:0x127e:0x0] : rc = -5
LustreError: 27868:0:(lov_object.c:1360:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 10114:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x200000403:0x151c:0x0] error -5.
LustreError: 19457:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '9' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 9' to finish migration: rc = -1
LustreError: 19457:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 23 previous similar messages
LustreError: 25230:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x12b4:0x0]: rc = -5
LustreError: 25230:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 164 previous similar messages
LustreError: 25230:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 25230:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 164 previous similar messages
13[5670]: segfault at 8 ip 00007f2fee4047e8 sp 00007ffff820a920 error 4 in ld-2.17.so[7f2fee3f9000+22000]
LustreError: 14867:0:(mdd_object.c:3873:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x13a6:0x0]: rc = -2
LustreError: 14867:0:(mdd_object.c:3873:mdd_close()) Skipped 1 previous similar message
1[12991]: segfault at 0 ip (null) sp 00007ffcce38d7a8 error 14 in 1[400000+6000]
19[14087]: segfault at 0 ip (null) sp 00007ffc7da9dd68 error 14 in 1[400000+6000]
LustreError: 10260:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x280000403:0x12d8:0x0] : rc = -5
LustreError: 10260:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
LustreError: 10260:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x280000403:0x12d8:0x0] error -5.
LustreError: 9972:0:(mdd_object.c:3873:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x185b:0x0]: rc = -2
LustreError: 9972:0:(mdd_object.c:3873:mdd_close()) Skipped 1 previous similar message
15[30745]: segfault at 8 ip 00007f058939a7e8 sp 00007ffff7fbc5b0 error 4 in ld-2.17.so[7f058938f000+22000]
LustreError: 24376:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x240000403:0x1743:0x0] error -5.
LustreError: 4466:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x14e3:0x0] ACL: rc = -2
Lustre: dir [0x240000403:0x15cd:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 161 previous similar messages
LustreError: 31233:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 2 [0x200000404:0x1564:0x0]: rc = -5
LustreError: 31233:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 2 previous similar messages
LustreError: 7187:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x200000404:0x1564:0x0] error -5.
LustreError: 7187:0:(vvp_io.c:1920:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 3497:0:(mdd_object.c:3873:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x1a27:0x0]: rc = -2
LustreError: 3497:0:(mdd_object.c:3873:mdd_close()) Skipped 1 previous similar message
LustreError: 9779:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x200000403:0x1a40:0x0] : rc = -5
LustreError: 9779:0:(lov_object.c:1360:lov_layout_change()) Skipped 9 previous similar messages
LustreError: 9779:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1a40:0x0] error -5.
LustreError: 26966:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x1b38:0x0] ACL: rc = -2
LustreError: 1410:0:(mdt_open.c:1280:mdt_cross_open()) lustre-MDT0001: [0x240000405:0xa04:0x0] doesn't exist!: rc = -14
LustreError: 20163:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x240000405:0x105c:0x0] error -5.
ptlrpc_watchdog_fire: 20 callbacks suppressed
Lustre: mdt04_003: service thread pid 26004 was inactive for 40.030 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 26004, comm: mdt04_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_check_lock+0xec/0x3c0 [mdt]
[<0>] mdt_object_stripes_lock+0xba/0x660 [mdt]
[<0>] mdt_reint_unlink+0x7a2/0x15e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbc0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt04_005: service thread pid 26425 was inactive for 40.092 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 1740, comm: mdt06_011 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf7/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1820 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt02_015: service thread pid 10911 was inactive for 40.031 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: Skipped 1 previous similar message
Pid: 26425, comm: mdt04_005 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf7/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1820 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc76/0x1690 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt06_003: service thread pid 26574 was inactive for 40.053 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_016: service thread pid 8586 was inactive for 40.033 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt03_004: service thread pid 26235 was inactive for 72.057 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
Lustre: mdt04_015: service thread pid 19339 was inactive for 74.208 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 15997:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88006864ad40/0x2e9add47b03b4761 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1966:0x0].0x0 bits 0x12/0x0 rrc: 14 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x2e9add47b03b4753 expref: 273 pid: 8612 timeout: 24446 lvb_type: 0
LustreError: 26235:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff88024f061bf8 ns: mdt-lustre-MDT0000_UUID lock: ffff880261fc7480/0x2e9add47b03b7471 lrc: 3/0,0 mode: PR/PR res: [0x200000400:0x6d:0x0].0x0 bits 0x12/0x0 rrc: 10 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2e9add47b03b7463 expref: 720 pid: 26235 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88027dd04a88: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff88027dd04a88: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88027dd04a88: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 32538:0:(file.c:5575:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x1966:0x0] error: rc = -5
LustreError: 31943:0:(vvp_io.c:1920:vvp_io_init()) lustre: refresh file layout [0x200000403:0x1dd6:0x0] error -5.
LustreError: 32557:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff88027dd04a88: [0x240000401:0x59:0x0] lock enqueue fails: rc = -108
LustreError: 32550:0:(llite_nfs.c:446:ll_dir_get_parent_fid()) lustre: failure inode [0x200000404:0x1a72:0x0] get parent: rc = -108
LustreError: 5129:0:(llite_lib.c:2019:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0001-mdc-ffff88027dd04a88: Connection restored to (at 0@lo)
Lustre: 20287:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000405:0x1193:0x0] with magic=0xbd60bd0
Lustre: 20287:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 57 previous similar messages
LustreError: 13232:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880270f05d28: cannot apply new layout on [0x240000403:0x1743:0x0] : rc = -5
LustreError: 13232:0:(lov_object.c:1360:lov_layout_change()) Skipped 5 previous similar messages

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mdt01_020:6927]
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk libata serio_raw i2c_core [last unloaded: libcfs]
CPU: 2 PID: 6927 Comm: mdt01_020 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8801d3598010 ti: ffff88013a78c000 task.ti: ffff88013a78c000
RIP: 0010:[<ffffffffa0cf5778>] [<ffffffffa0cf5778>] zap_leaf_lookup+0xc8/0x160 [zfs]
RSP: 0018:ffff88013a78f400 EFLAGS: 00000207
RAX: 0000000000000000 RBX: ffff88013a78f390 RCX: 0000000000000009
RDX: ffff88028db79430 RSI: 0000000000000200 RDI: bfb889c75cb10000
RBP: ffff88013a78f430 R08: 0000000000000000 R09: ffff88028db79000
R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
R13: ffff88013a78f468 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000141143c CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0cf355c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c56acd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0cf3651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0cf8dfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0cf7a97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0cf9085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa0a380a8>] osd_dir_insert+0x228/0x9b0 [osd_zfs]
[<ffffffffa13cb0ed>] lod_sub_insert+0x1cd/0x3b0 [lod]
[<ffffffffa13b2cdf>] ? lod_create+0x28f/0x3b0 [lod]
[<ffffffffa12310e1>] ? mdd_create_object_internal+0xb1/0x360 [mdd]
[<ffffffffa1398816>] lod_insert+0x16/0x20 [lod]
[<ffffffffa1213126>] __mdd_index_insert_only+0xc6/0x2d0 [mdd]
[<ffffffffa1215ee9>] __mdd_index_insert+0x49/0x2e0 [mdd]
[<ffffffffa121ed4d>] mdd_create+0x117d/0x1e30 [mdd]
[<ffffffffa12c93d3>] mdt_reint_open+0x21c3/0x2e40 [mdt]
[<ffffffffa12b9817>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa128e74c>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<ffffffffa129aaa3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa12928c9>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<ffffffffa129aa10>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa129856a>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa067c221>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<ffffffffa01a1dc5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa01a5036>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa06a61db>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<ffffffffa0622200>] ? lustre_msg_buf_v2+0x150/0x1f0 [ptlrpc]
[<ffffffffa06e7fc8>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa06ee24e>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<ffffffffa06348bc>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<ffffffffa0636656>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffffa06359f0>] ? ptlrpc_wait_event+0x610/0x610 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 3481:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800799bdf40 x1795473566626816/t4294967594(0) o101->a07a486a-a291-430c-b0e6-f04aff39e4f5@0@lo:166/0 lens 376/864 e 0 to 0 dl 1712297131 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 20971:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x280000403:0x2a:0x0] mdc close failed: rc = -13
Lustre: dir [0x240000404:0x1d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 20265:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x280000403:0x17:0x0] mdc close failed: rc = -2
LustreError: 20551:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/9 failed: rc = -16
LustreError: 3528:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/8 failed: rc = -16
LustreError: 22874:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e012548: inode [0x200000403:0xd1:0x0] mdc close failed: rc = -116
LustreError: 22874:0:(file.c:264:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 3545:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/2 failed: rc = -114
LustreError: 3545:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 1 previous similar message
Lustre: dir [0x240000403:0x29:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 24484:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x240000404:0x4a:0x0] mdc close failed: rc = -116
LustreError: 24484:0:(file.c:264:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 13422:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009c461940 x1795473568012544/t4294968660(0) o101->51fe59a9-777a-4777-94d5-0d541770d6e0@0@lo:194/0 lens 376/864 e 0 to 0 dl 1712297159 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 3545:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/13 failed: rc = -16
LustreError: 3545:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 6 previous similar messages
Lustre: 3470:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8801855b4640 x1795473569326848/t4294970272(0) o101->a07a486a-a291-430c-b0e6-f04aff39e4f5@0@lo:219/0 lens 376/864 e 0 to 0 dl 1712297184 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: mdt06_007: service thread pid 21284 was inactive for 40.063 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 21284, comm: mdt06_007 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_object_open_lock+0x669/0xb50 [mdt]
[<0>] mdt_reint_open+0x24cb/0x2e40 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_intent_open+0x93/0x480 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22969, comm: mdt_io06_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_migrate_links_lock+0x85e/0x134e [mdt]
[<0>] mdt_reint_migrate+0xdc0/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 3548, comm: mdt_io07_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x377/0x420 [mdt]
[<0>] mdt_rename_lock+0xc3/0x2d0 [mdt]
[<0>] mdt_reint_migrate+0x891/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io00_003: service thread pid 27026 was inactive for 40.117 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io06_001: service thread pid 3545 was inactive for 40.147 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io05_003: service thread pid 23092 was inactive for 40.023 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: mdt_io01_002: service thread pid 3531 was inactive for 40.095 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt_io03_001: service thread pid 3536 was inactive for 40.098 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt_io04_001: service thread pid 3539 was inactive for 72.226 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 3369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880276f36940/0xa7df3fff054d211f lrc: 3/0,0 mode: PW/PW res: [0x200000403:0x112:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa7df3fff054d2111 expref: 205 pid: 21284 timeout: 11967 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88009e012548: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff88009e012548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 22969:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/12 failed: rc = -16
LustreError: 22969:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 2 previous similar messages
LustreError: 167-0: lustre-MDT0000-mdc-ffff88009e012548: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 25946:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff88009e012548: [0x200000403:0x1:0x0] lock enqueue fails: rc = -108
LustreError: 23214:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e012548: inode [0x200000403:0x112:0x0] mdc close failed: rc = -5
Lustre: dir [0x200000404:0xe5:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 15 previous similar messages
LustreError: 30383:0:(llite_lib.c:2019:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0000-mdc-ffff88009e012548: Connection restored to (at 0@lo)
LustreError: 28942:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '4' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 4' to finish migration: rc = -1
LustreError: 3528:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/5 failed: rc = -16
LustreError: 3528:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 10 previous similar messages
LustreError: 26116:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e012548: inode [0x200000403:0x5:0x0] mdc close failed: rc = -2
LustreError: 26116:0:(file.c:264:ll_close_inode_openhandle()) Skipped 11 previous similar messages
0[2710]: segfault at 8 ip 00007f2e8d34f7e8 sp 00007ffd0395bf50 error 4 in ld-2.17.so[7f2e8d344000+22000]
LustreError: 5524:0:(llite_lib.c:1868:ll_update_lsm_md()) lustre: [0x280000403:0xad:0x0] dir layout mismatch:
LustreError: 5524:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x2 count=2 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 5524:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) stripe[0] [0x280000400:0xf:0x0]
LustreError: 5524:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=3 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
Lustre: dir [0x240000403:0x9a:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 7297:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x2 count=2 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 7297:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=3 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 6295:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0x200:0x0]: rc = -5
LustreError: 6295:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 3526:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000403:0x17a:0x0]/10 failed: rc = -2
LustreError: 3526:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 20 previous similar messages
LustreError: 23092:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '3' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 3' to finish migration: rc = -1
LustreError: 26589:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x280000403:0x36:0x0] mdc close failed: rc = -2
LustreError: 26589:0:(file.c:264:ll_close_inode_openhandle()) Skipped 4 previous similar messages
LustreError: 29079:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '10' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 10' to finish migration: rc = -1
LustreError: 16197:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xae:0x0]: rc = -5
LustreError: 16197:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
Lustre: dir [0x200000405:0xad:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
LustreError: 1497:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 0' to finish migration: rc = -1
LustreError: 3536:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/11 failed: rc = -114
LustreError: 3536:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 23 previous similar messages
LustreError: 24807:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x254:0x0]: rc = -5
LustreError: 24807:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 24807:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 24807:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 1 previous similar message
Lustre: 21321:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x295:0x0] with magic=0xbd60bd0
LustreError: 18555:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '3' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 3' to finish migration: rc = -1
LustreError: 23071:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x297:0x0]: rc = -5
LustreError: 23071:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 24819:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x254:0x0]: rc = -5
LustreError: 24819:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 18555:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 0' to finish migration: rc = -1
LustreError: 26246:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000403:0xae:0x0]: rc = -5
LustreError: 26246:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 3 previous similar messages
LustreError: 26246:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 26246:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 3 previous similar messages
Lustre: 25848:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x1bb:0x0] with magic=0xbd60bd0
Lustre: 25848:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 27 previous similar messages
Lustre: 21999:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x254:0x0] with magic=0xbd60bd0
Lustre: 21999:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
2[32505]: segfault at 8 ip 00007f5bb815d7e8 sp 00007fffdf4b1a80 error 4 in ld-2.17.so[7f5bb8152000+22000]
LustreError: 27956:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '4' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 4' to finish migration: rc = -1
LustreError: 27956:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 1420:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x379:0x0]: rc = -5
LustreError: 1420:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 1420:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 1420:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 2904:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x280000404:0x232:0x0] mdc close failed: rc = -2
LustreError: 2904:0:(file.c:264:ll_close_inode_openhandle()) Skipped 3 previous similar messages
Lustre: dir [0x280000404:0x26b:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 14 previous similar messages
LustreError: 29079:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/2 failed: rc = -16
LustreError: 29079:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 43 previous similar messages
LustreError: 11789:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000404:0x354:0x0]: rc = -5
LustreError: 11789:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 9 previous similar messages
LustreError: 11789:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 11789:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 9 previous similar messages
LustreError: 4035:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '14' migration was interrupted, run 'lfs migrate -m 1 -c 2 -H crush 14' to finish migration: rc = -1
LustreError: 4035:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
Lustre: 23385:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x287:0x0] with magic=0xbd60bd0
Lustre: 23385:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
Lustre: dir [0x280000404:0x3ab:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x200000404:0x5cb:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 9 previous similar messages
Lustre: 21602:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x51f:0x0] with magic=0xbd60bd0
Lustre: 21602:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 28644:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x379:0x0]: rc = -5
LustreError: 28644:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 9 previous similar messages
LustreError: 28644:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 28644:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 9 previous similar messages
LustreError: 31268:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x200000404:0x710:0x0] mdc close failed: rc = -116
LustreError: 31268:0:(file.c:264:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 3540:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '2' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 2' to finish migration: rc = -1
LustreError: 3540:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 1 previous similar message
10[5702]: segfault at 8 ip 00007fe052b357e8 sp 00007ffefd197d00 error 4 in ld-2.17.so[7fe052b2a000+22000]
LustreError: 1497:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x5da:0x0]/5 failed: rc = -114
LustreError: 1497:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 69 previous similar messages
LustreError: 3486:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x67e:0x0] migrate mdt count mismatch 1 != 2
LustreError: 21972:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x83c:0x0] migrate mdt count mismatch 1 != 3
Lustre: dir [0x280000404:0x6b5:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
1[27790]: segfault at 8 ip 00007fd91703d7e8 sp 00007ffd7000a960 error 4 in ld-2.17.so[7fd917032000+22000]
LustreError: 26341:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for sleep [0x240000404:0x4d9:0x0]: rc = -5
LustreError: 28735:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x877:0x0]: rc = -5
LustreError: 28735:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 19 previous similar messages
LustreError: 28735:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 28735:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 19 previous similar messages
LustreError: 28735:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 12 [0x200000404:0x877:0x0]: rc = -5
Lustre: 25842:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x5a8:0x0] with magic=0xbd60bd0
Lustre: 25842:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 15 previous similar messages
ptlrpc_watchdog_fire: 17 callbacks suppressed
Lustre: mdt02_012: service thread pid 23385 was inactive for 40.097 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 23385, comm: mdt02_012 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xbf3/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 13522, comm: mdt01_012 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xbf3/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 11233, comm: mdt02_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Lustre: mdt01_018: service thread pid 13913 was inactive for 40.126 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 8 previous similar messages
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xbf3/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt05_012: service thread pid 25831 was inactive for 72.158 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
LustreError: 3369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8801c1033100/0xa7df3fff056e4709 lrc: 3/0,0 mode: PW/PW res: [0x280000404:0x868:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa7df3fff056e46df expref: 213 pid: 26477 timeout: 12400 lvb_type: 0
LustreError: 11-0: lustre-MDT0002-mdc-ffff88032519ca88: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff88032519ca88: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 13522:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff88009c0e0008 ns: mdt-lustre-MDT0002_UUID lock: ffff88009c174780/0xa7df3fff056e94f0 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT gid 0 flags: 0x50200400000020 nid: 0@lo remote: 0xa7df3fff056e9464 expref: 23 pid: 13522 timeout: 0 lvb_type: 0
LustreError: 167-0: lustre-MDT0002-mdc-ffff88032519ca88: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: 30696:0:(file.c:5550:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -5
LustreError: 19314:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88032519ca88: inode [0x280000404:0x7d7:0x0] mdc close failed: rc = -108
LustreError: 19314:0:(file.c:264:ll_close_inode_openhandle()) Skipped 8 previous similar messages
LustreError: 30696:0:(file.c:5550:ll_inode_revalidate_fini()) Skipped 11 previous similar messages
LustreError: 17211:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0002-mdc-ffff88032519ca88: [0x280000403:0x33e:0x0] lock enqueue fails: rc = -108
LustreError: 17211:0:(mdc_request.c:1469:mdc_read_page()) Skipped 7 previous similar messages
LustreError: 22868:0:(ldlm_resource.c:1127:ldlm_resource_complain()) lustre-MDT0002-mdc-ffff88032519ca88: namespace resource [0x240000403:0x1:0x0].0x0 (ffff88025c53f9c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0002-mdc-ffff88032519ca88: Connection restored to (at 0@lo)
Lustre: 3472:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x4d9:0x0] with magic=0xbd60bd0
Lustre: 3472:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 4035:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '9' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 9' to finish migration: rc = -1
LustreError: 4035:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
19[26505]: segfault at 0 ip (null) sp 00007ffe95b63618 error 14 in 17[400000+6000]
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000404:0x8a4:0x0]/0xa): rc = 0
Lustre: 26318:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x9c5:0x0] with magic=0xbd60bd0
Lustre: 26318:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 9 previous similar messages
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000404:0xa57:0x0]/0xa): rc = 0
LustreError: 6609:0:(osd_index.c:221:__osd_xattr_load_by_oid()) lustre-MDT0000: can't get bonus, rc = -2
LustreError: 3538:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/15 failed: rc = -114
LustreError: 3538:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 108 previous similar messages
LustreError: 28735:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 19 [0x240000403:0xcc3:0x0]: rc = -5
LustreError: 28735:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 1 previous similar message
Lustre: dir [0x200000404:0xa7d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
LustreError: 10863:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for sleep [0x200000405:0xa73:0x0]: rc = -5
LustreError: 3469:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0x84b:0x0] migrate mdt count mismatch 3 != 1
LustreError: 3490:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0xa35:0x0]: rc = -2
LustreError: 20651:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88032519ca88: cannot apply new layout on [0x200000405:0xa73:0x0] : rc = -5
LustreError: 20651:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x200000405:0xa73:0x0] error -5.
LustreError: 7899:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 19 [0x240000403:0xcc3:0x0]: rc = -5
LustreError: 3491:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0xa35:0x0]: rc = -2
Lustre: 12857:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0xe52:0x0] with magic=0xbd60bd0
Lustre: 12857:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 9 previous similar messages
LustreError: 20257:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88032519ca88: cannot apply new layout on [0x200000405:0xa73:0x0] : rc = -5
LustreError: 20257:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0xa73:0x0]: rc = -5
LustreError: 20257:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 20 previous similar messages
LustreError: 20257:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 20257:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 20 previous similar messages
LustreError: 20257:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for sleep [0x200000405:0xa73:0x0]: rc = -5
12[30514]: segfault at 8 ip 00007f59e422d7e8 sp 00007ffd6a800420 error 4 in ld-2.17.so[7f59e4222000+22000]
LustreError: 15340:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0002: [0x280000405:0x3d2:0x0] migrate mdt count mismatch 1 != 2
LustreError: 23663:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 17 [0x200000404:0xec9:0x0]: rc = -5
2[28659]: segfault at 8 ip 00007f0314b667e8 sp 00007ffe739d1b90 error 4 in ld-2.17.so[7f0314b5b000+22000]
LustreError: 3532:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '6' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 6' to finish migration: rc = -1
LustreError: 3532:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 16 previous similar messages
Lustre: 13320:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0xc34:0x0] with magic=0xbd60bd0
Lustre: 13320:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 9 previous similar messages
LustreError: 5271:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e012548: inode [0x280000403:0xc44:0x0] mdc close failed: rc = -116
LustreError: 5271:0:(file.c:264:ll_close_inode_openhandle()) Skipped 40 previous similar messages
LustreError: 10938:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88009e012548: cannot apply new layout on [0x200000404:0xe54:0x0] : rc = -5
LustreError: 10938:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
LustreError: 10938:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x200000404:0xe54:0x0] error -5.
LustreError: 16822:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88009e012548: cannot apply new layout on [0x200000404:0xe54:0x0] : rc = -5
LustreError: 18082:0:(llite_lib.c:1868:ll_update_lsm_md()) lustre: [0x280000403:0xcd0:0x0] dir layout mismatch:
LustreError: 18082:0:(llite_lib.c:1868:ll_update_lsm_md()) Skipped 1 previous similar message
LustreError: 18082:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x2 count=2 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 18082:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) stripe[0] [0x280000400:0x54:0x0]
LustreError: 18082:0:(lustre_lmv.h:185:lmv_stripe_object_dump()) Skipped 9 previous similar messages
LustreError: 18082:0:(lustre_lmv.h:178:lmv_stripe_object_dump()) dump LMV: refs 215092432 magic=0x1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=crush:3 pool=
LustreError: 26742:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88009e012548: cannot apply new layout on [0x200000404:0xe54:0x0] : rc = -5
LustreError: 30851:0:(llite_nfs.c:446:ll_dir_get_parent_fid()) lustre: failure inode [0x280000403:0xd28:0x0] get parent: rc = -2
LustreError: 26006:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88032519ca88: cannot apply new layout on [0x200000405:0x11c6:0x0] : rc = -5
LustreError: 26006:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
LustreError: 26006:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x200000405:0x11c6:0x0] error -5.
LustreError: 3464:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x140a:0x0] migrate mdt count mismatch 1 != 2
LustreError: 12857:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x2b87:0x0] ACL: rc = -2
Lustre: dir [0x280000403:0xec8:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 27 previous similar messages
11[20879]: segfault at 8 ip 00007f08e81d27e8 sp 00007ffd3fb201b0 error 4 in ld-2.17.so[7f08e81c7000+22000]
18[27019]: segfault at 8 ip 00007f2c8339e7e8 sp 00007ffd895c2ec0 error 4 in ld-2.17.so[7f2c83393000+22000]
16[3153]: segfault at 0 ip (null) sp 00007ffd7f28d5d8 error 14 in 16[400000+6000]
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000404:0x2d89:0x0]/0xa): rc = 0
8[11342]: segfault at 8 ip 00007fbce26517e8 sp 00007ffc4b3221a0 error 4 in ld-2.17.so[7fbce2646000+22000]
Lustre: 32632:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x180a:0x0] with magic=0xbd60bd0
Lustre: 32632:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 73 previous similar messages
ptlrpc_watchdog_fire: 7 callbacks suppressed
Lustre: mdt05_016: service thread pid 23009 was inactive for 40.128 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 23009, comm: mdt05_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_object_open_lock+0x669/0xb50 [mdt]
[<0>] mdt_reint_open+0x24cb/0x2e40 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_intent_open+0x93/0x480 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io05_003: service thread pid 23092 was inactive for 74.292 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 23092, comm: mdt_io05_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x4d9/0x7e0 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_reint_migrate+0xd68/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io05_005: service thread pid 18555 was inactive for 74.035 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 18555, comm: mdt_io05_005 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x377/0x420 [mdt]
[<0>] mdt_rename_lock+0xc3/0x2d0 [mdt]
[<0>] mdt_reint_migrate+0x891/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io02_004: service thread pid 4448 was inactive for 76.095 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff88032519ca88: disconnect after 24s idle
LustreError: 3369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8801e6add2c0/0xa7df3fff05a3d6ab lrc: 3/0,0 mode: PW/PW res: [0x200000405:0x15b0:0x0].0x0 bits 0x4/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa7df3fff05a3d688 expref: 242 pid: 13208 timeout: 13049 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff88009e012548: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 8 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88009e012548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff88009e012548: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 19349:0:(llite_lib.c:2019:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 19349:0:(llite_lib.c:2019:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 14621:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0000-mdc-ffff88009e012548: [0x200000404:0x16df:0x0] lock enqueue fails: rc = -108
LustreError: 14621:0:(mdc_request.c:1469:mdc_read_page()) Skipped 5 previous similar messages
LustreError: 17453:0:(file.c:5550:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 17453:0:(file.c:5550:ll_inode_revalidate_fini()) Skipped 44 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88009e012548: Connection restored to (at 0@lo)
LustreError: 3546:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/4 failed: rc = -114
LustreError: 3546:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 270 previous similar messages
LustreError: 28041:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x10b7:0x0]: rc = -5
LustreError: 28041:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 193 previous similar messages
LustreError: 28041:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 28041:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 193 previous similar messages
LustreError: 25965:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88009e012548: cannot apply new layout on [0x280000405:0xb53:0x0] : rc = -5
LustreError: 25965:0:(lov_object.c:1360:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 25965:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x280000405:0xb53:0x0] error -5.
LustreError: 31036:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000403:0x194d:0x0] error -5.
LustreError: 15905:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 10 [0x240000403:0x1a2e:0x0]: rc = -5
15[24909]: segfault at 8 ip 00007f8629fe27e8 sp 00007ffd280c9d80 error 4 in ld-2.17.so[7f8629fd7000+22000]
LustreError: 28751:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x1a6e:0x0] migrate mdt count mismatch 1 != 3
LustreError: 16875:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 0' to finish migration: rc = -1
LustreError: 16875:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 31 previous similar messages
14[15034]: segfault at 0 ip (null) sp 00007ffd06f18668 error 14 in 14[400000+6000]
LustreError: 23404:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x3234:0x0] migrate mdt count mismatch 2 != 3
LustreError: 20504:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e012548: inode [0x200000404:0x1c60:0x0] mdc close failed: rc = -13
LustreError: 20504:0:(file.c:264:ll_close_inode_openhandle()) Skipped 36 previous similar messages
LustreError: 26341:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 11 [0x280000403:0x1311:0x0]: rc = -5
LustreError: 28751:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x14c5:0x0] ACL: rc = -2
LustreError: 28751:0:(mdt_handler.c:777:mdt_pack_acl2body()) Skipped 1 previous similar message
LustreError: 9137:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88032519ca88: cannot apply new layout on [0x240000403:0x205b:0x0] : rc = -5
LustreError: 9137:0:(lov_object.c:1360:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 9137:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000403:0x205b:0x0] error -5.
19[25730]: segfault at 4017b0 ip 00000000004017b0 sp 00007ffe78de2fd0 error 14
ptlrpc_watchdog_fire: 16 callbacks suppressed
Lustre: mdt06_018: service thread pid 26789 was inactive for 74.114 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 26789, comm: mdt06_018 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_object_find_lock+0x54/0x170 [mdt]
[<0>] mdt_reint_setxattr+0x133/0x1020 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 21321, comm: mdt06_008 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock_try+0xa0/0x250 [mdt]
[<0>] mdt_getattr_name_lock+0x1699/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 4727:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 1 [0x240000403:0x2261:0x0]: rc = -5
LustreError: 4727:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 3 previous similar messages
LustreError: 3369:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88025d8cd680/0xa7df3fff05c24fde lrc: 3/0,0 mode: CR/CR res: [0x240000403:0x20e9:0x0].0x0 bits 0xa/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xa7df3fff05c24f83 expref: 445 pid: 12932 timeout: 13384 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff88009e012548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff88009e012548: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff88009e012548: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 8450:0:(mdc_request.c:1469:mdc_read_page()) lustre-MDT0001-mdc-ffff88009e012548: [0x240000402:0x40:0x0] lock enqueue fails: rc = -108
LustreError: 8450:0:(mdc_request.c:1469:mdc_read_page()) Skipped 13 previous similar messages
Lustre: dir [0x280000405:0x1302:0x0] stripe 2 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 100 previous similar messages
LustreError: 15024:0:(file.c:5550:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x20e9:0x0] error: rc = -108
LustreError: 15024:0:(file.c:5550:ll_inode_revalidate_fini()) Skipped 5 previous similar messages
LustreError: 11123:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000403:0x20e9:0x0] error -108.
LustreError: 18058:0:(ldlm_resource.c:1127:ldlm_resource_complain()) lustre-MDT0001-mdc-ffff88009e012548: namespace resource [0x240000404:0x3234:0x0].0x0 (ffff88016db10a40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 18058:0:(ldlm_resource.c:1127:ldlm_resource_complain()) Skipped 2 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff88009e012548: Connection restored to (at 0@lo)
LustreError: 31186:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff88008ad1b7e8 ns: mdt-lustre-MDT0001_UUID lock: ffff88005e429a80/0xa7df3fff05cc8f87 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x69:0x0].0x0 bits 0x12/0x0 rrc: 24 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xa7df3fff05cc8f79 expref: 2 pid: 31186 timeout: 0 lvb_type: 0
LustreError: 31186:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) Skipped 7 previous similar messages
LustreError: 4264:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff88032519ca88: cannot apply new layout on [0x240000403:0x1fc4:0x0] : rc = -5
LustreError: 4264:0:(lov_object.c:1360:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 4264:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000403:0x1fc4:0x0] error -5.
1[14996]: segfault at 0 ip (null) sp 00007ffd3141aa18 error 14 in 1[400000+6000]
Lustre: 11233:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000406:0xe07:0x0] with magic=0xbd60bd0
Lustre: 11233:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 127 previous similar messages

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [mdt00_024:15969]
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix serio_raw floppy virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 1 PID: 15969 Comm: mdt00_024 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802cd1d5c40 ti: ffff880182a40000 task.ti: ffff880182a40000
RIP: 0010:[<ffffffffa0ced73c>] [<ffffffffa0ced73c>] zap_leaf_lookup+0x8c/0x160 [zfs]
RSP: 0018:ffff880182a43400 EFLAGS: 00000213
RAX: 0000000000000000 RBX: ffff880182a43390 RCX: 0000000000000009
RDX: ffff8802ca749430 RSI: 0000000000000200 RDI: 63fc8675adc40000
RBP: ffff880182a43430 R08: 0000000000000000 R09: ffff8802ca749000
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
R13: ffff880182a43468 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe8eb00f9e5 CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0ceb55c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c4eacd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0ceb651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0cf0dfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0cefa97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0cf1085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa11740a8>] osd_dir_insert+0x228/0x9b0 [osd_zfs]
[<ffffffffa13d80ed>] lod_sub_insert+0x1cd/0x3b0 [lod]
[<ffffffffa13bfcdf>] ? lod_create+0x28f/0x3b0 [lod]
[<ffffffffa123e0e1>] ? mdd_create_object_internal+0xb1/0x360 [mdd]
[<ffffffffa13a5816>] lod_insert+0x16/0x20 [lod]
[<ffffffffa1220126>] __mdd_index_insert_only+0xc6/0x2d0 [mdd]
[<ffffffffa1222ee9>] __mdd_index_insert+0x49/0x2e0 [mdd]
[<ffffffffa122bd4d>] mdd_create+0x117d/0x1e30 [mdd]
[<ffffffffa12d63d3>] mdt_reint_open+0x21c3/0x2e40 [mdt]
[<ffffffffa12c6817>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa129b74c>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<ffffffffa12a7aa3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa129f8c9>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<ffffffffa12a7a10>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa12a556a>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa0680221>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<ffffffffa016fdc5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa0173036>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa06aa1db>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<ffffffffa0626200>] ? lustre_msg_buf_v2+0x150/0x1f0 [ptlrpc]
[<ffffffffa06ebfc8>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa06f224e>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<ffffffffa06388bc>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<ffffffffa063a656>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<ffffffff810dbb51>] ? put_prev_entity+0x31/0x400
[<ffffffffa06399f0>] ? ptlrpc_wait_event+0x610/0x610 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

LustreError: 14103:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x28:0x0] ACL: rc = -2
LustreError: 32367:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/4 failed: rc = -16
LustreError: 20324:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x200000403:0x6c:0x0] mdc close failed: rc = -116
Lustre: 9792:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009e30da40 x1794733882388544/t4294969091(0) o101->6e55a4cd-4564-4f08-aa78-9fad6ccf54a4@0@lo:709/0 lens 376/864 e 0 to 0 dl 1711591749 ref 1 fl Interpret:H/202/0 rc 0/0 job:'cat.0' uid:0 gid:0
Lustre: 32311:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009c8e3240 x1794733882391104/t4294968532(0) o101->6e55a4cd-4564-4f08-aa78-9fad6ccf54a4@0@lo:710/0 lens 376/864 e 0 to 0 dl 1711591750 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 32375:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/8 failed: rc = -114
LustreError: 21715:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x200000403:0xdc:0x0] mdc close failed: rc = -116
LustreError: 22400:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x200000404:0x91:0x0] mdc close failed: rc = -116
LustreError: 32369:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/18 failed: rc = -114
LustreError: 32369:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 1 previous similar message
LustreError: 32383:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0002: migrate [0x280000403:0x1:0x0]/8 failed: rc = -114
LustreError: 32383:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 3 previous similar messages
LustreError: 32387:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '13' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 13' to finish migration: rc = -1
LustreError: 23393:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x240000404:0x103:0x0] mdc close failed: rc = -116
LustreError: 23393:0:(file.c:264:ll_close_inode_openhandle()) Skipped 4 previous similar messages
LustreError: 23833:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/15 failed: rc = -114
LustreError: 23833:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 9 previous similar messages
LustreError: 22823:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x200000404:0x6:0x0] mdc close failed: rc = -116
LustreError: 22823:0:(file.c:264:ll_close_inode_openhandle()) Skipped 3 previous similar messages
LustreError: 32387:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '2' migration was interrupted, run 'lfs migrate -m 1 -c 3 -H crush 2' to finish migration: rc = -1
LustreError: 22773:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x240000403:0x159:0x0]/17 failed: rc = -114
LustreError: 22773:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 5 previous similar messages
0[26850]: segfault at 8 ip 00007f6fd83bb7e8 sp 00007ffc0c593610 error 4 in ld-2.17.so[7f6fd83b0000+22000]
Lustre: 9366:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0xe3:0x0] with magic=0xbd60bd0
12[27896]: segfault at 8 ip 00007feb9a6bf7e8 sp 00007ffdda986ca0 error 4 in ld-2.17.so[7feb9a6b4000+22000]
LustreError: 28770:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x280000404:0xd7:0x0] mdc close failed: rc = -116
LustreError: 28770:0:(file.c:264:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 29723:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x1bf:0x0]: rc = -5
LustreError: 29723:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 32365:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x166:0x0]/3 failed: rc = -114
LustreError: 32365:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 16 previous similar messages
Lustre: 11254:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802ae3ff840 x1794733885396096/t4294971705(0) o101->6e55a4cd-4564-4f08-aa78-9fad6ccf54a4@0@lo:15/0 lens 376/816 e 0 to 0 dl 1711591810 ref 1 fl Interpret:H/202/0 rc 0/0 job:'dd.0' uid:0 gid:0
LustreError: 3872:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x200000404:0x1db:0x0] mdc close failed: rc = -116
LustreError: 3872:0:(file.c:264:ll_close_inode_openhandle()) Skipped 3 previous similar messages
LustreError: 18927:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0001: migrate [0x240000403:0x1:0x0]/4 failed: rc = -16
LustreError: 18927:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 21 previous similar messages
Lustre: dir [0x280000404:0x15b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 32313:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x183:0x0] with magic=0xbd60bd0
Lustre: 32313:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 16115:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x240000404:0x2c3:0x0] mdc close failed: rc = -116
LustreError: 16115:0:(file.c:264:ll_close_inode_openhandle()) Skipped 5 previous similar messages
LustreError: 14619:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x1a0:0x0]: rc = -5
LustreError: 14619:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
Lustre: dir [0x240000403:0x1dd:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: 10452:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x1f1:0x0] with magic=0xbd60bd0
Lustre: 10452:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 12103:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x437:0x0] with magic=0xbd60bd0
Lustre: 12103:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 23982:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x358:0x0]: rc = -5
LustreError: 23982:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 23982:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 23982:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 32385:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '9' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 9' to finish migration: rc = -1
Lustre: 29270:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x39a:0x0] with magic=0xbd60bd0
Lustre: 29270:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 32381:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/12 failed: rc = -114
LustreError: 32381:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 43 previous similar messages
LustreError: 18927:0:(lustre_lmv.h:517:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=4 index=3 hash=crush:0x82000003 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 32319:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x492:0x0] migrate mdt count mismatch 3 != 2
LustreError: 32366:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '6' migration was interrupted, run 'lfs migrate -m 2 -c 3 -H crush 6' to finish migration: rc = -1
Lustre: 32307:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x293:0x0] with magic=0xbd60bd0
Lustre: 32307:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 2198:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x506:0x0]: rc = -5
LustreError: 2198:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 5 previous similar messages
LustreError: 2198:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 2198:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 5 previous similar messages
LustreError: 32385:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '15' migration was interrupted, run 'lfs migrate -m 2 -c 2 -H crush 15' to finish migration: rc = -1
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000404:0x33c:0x0]/0xa): rc = 0
LustreError: 3160:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x200000403:0x418:0x0] mdc close failed: rc = -116
LustreError: 3160:0:(file.c:264:ll_close_inode_openhandle()) Skipped 6 previous similar messages
LustreError: 30562:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '9' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 9' to finish migration: rc = -1
LustreError: 30562:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 1046:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000404:0x504:0x0]: rc = -5
LustreError: 1046:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
Lustre: dir [0x200000404:0x492:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: 9792:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x413:0x0] with magic=0xbd60bd0
Lustre: 9792:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 7 previous similar messages
LustreError: 10571:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x5d5:0x0]: rc = -5
LustreError: 10571:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 10571:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 10571:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 10475:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '16' migration was interrupted, run 'lfs migrate -m 0 -c 2 -H crush 16' to finish migration: rc = -1
LustreError: 10475:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 1 previous similar message
14[12214]: segfault at 8 ip 00007f69e1d3c7e8 sp 00007ffd4c22aa40 error 4 in ld-2.17.so[7f69e1d31000+22000]
Lustre: 10211:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x456:0x0] with magic=0xbd60bd0
Lustre: 10211:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 14861:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x3bf:0x0]: rc = -5
LustreError: 14861:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 4762:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x240000404:0x506:0x0] : rc = -5
LustreError: 4762:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000404:0x506:0x0] error -5.
Lustre: dir [0x280000404:0x33b:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 23719:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0000: '16' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 16' to finish migration: rc = -1
LustreError: 24779:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x714:0x0]: rc = -5
LustreError: 24779:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 9 previous similar messages
LustreError: 24779:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 24779:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 9 previous similar messages
LustreError: 24044:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x1:0x0]/12 failed: rc = -16
LustreError: 24044:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 73 previous similar messages
Lustre: dir [0x200000403:0x4ab:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 21183:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 3 [0x200000404:0x714:0x0]: rc = -5
LustreError: 24805:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x240000403:0x3ff:0x0] mdc close failed: rc = -2
LustreError: 24805:0:(file.c:264:ll_close_inode_openhandle()) Skipped 8 previous similar messages
Lustre: dir [0x280000403:0x3fb:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
Lustre: 27051:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x46d:0x0] with magic=0xbd60bd0
Lustre: 27051:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 5 previous similar messages
LustreError: 32380:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '17' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 17' to finish migration: rc = -1
LustreError: 32380:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 1 previous similar message
LustreError: 23588:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for sleep [0x200000404:0x3bf:0x0]: rc = -5
LustreError: 13630:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0x46c:0x0]: rc = -5
LustreError: 13630:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 24 previous similar messages
LustreError: 13630:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 13630:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 24 previous similar messages
LustreError: 13630:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 8 [0x280000403:0x46c:0x0]: rc = -5
LustreError: 13630:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 1 previous similar message
10[10844]: segfault at 8 ip 00007fed70f2d7e8 sp 00007ffce7c63080 error 4 in ld-2.17.so[7fed70f22000+22000]
3[21556]: segfault at 8 ip 00007fa328c397e8 sp 00007ffffcc58520 error 4 in ld-2.17.so[7fa328c2e000+22000]
Lustre: dir [0x280000403:0x6bd:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
8[27624]: segfault at 8 ip 00007f91e430e7e8 sp 00007ffdf978a2b0 error 4 in ld-2.17.so[7f91e4303000+22000]
Lustre: 27746:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x615:0x0] with magic=0xbd60bd0
Lustre: 27746:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 39 previous similar messages
8[1685]: segfault at 8 ip 00007fb79cbf87e8 sp 00007ffc2b41fd00 error 4 in ld-2.17.so[7fb79cbed000+22000]
LustreError: 32381:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 15' to finish migration: rc = -1
LustreError: 32381:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 9 previous similar messages
LustreError: 3844:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x8df:0x0]: rc = -5
LustreError: 3844:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 52 previous similar messages
LustreError: 3844:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 3844:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 52 previous similar messages
Lustre: dir [0x240000403:0x958:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
0[23175]: segfault at 8 ip 00007f4c0d5217e8 sp 00007ffec1829440 error 4 in ld-2.17.so[7f4c0d516000+22000]
LustreError: 32387:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x280000403:0x6d4:0x0]/5 failed: rc = -2
LustreError: 32387:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 161 previous similar messages
LustreError: 6237:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff880264586678: inode [0x240000403:0xa1f:0x0] mdc close failed: rc = -116
LustreError: 6237:0:(file.c:264:ll_close_inode_openhandle()) Skipped 17 previous similar messages
LustreError: 32341:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xbb2:0x0]: rc = -2
LustreError: 16359:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xbb2:0x0]: rc = -2
Lustre: dir [0x280000404:0xc06:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
LustreError: 10667:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0xf30:0x0] migrate mdt count mismatch 3 != 2
LustreError: 7207:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0xda1:0x0] migrate mdt count mismatch 2 != 3
LustreError: 30316:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x280000404:0xc9f:0x0] : rc = -5
LustreError: 30316:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
LustreError: 30316:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x280000404:0xc9f:0x0] error -5.
LustreError: 10539:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x280000404:0xc9f:0x0] : rc = -5
Lustre: 32319:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0xf46:0x0] with magic=0xbd60bd0
Lustre: 32319:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 27 previous similar messages
LustreError: 27469:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0002: [0x280000404:0x1061:0x0] migrate mdt count mismatch 3 != 1
LustreError: 32377:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '8' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 8' to finish migration: rc = -1
LustreError: 32377:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 22 previous similar messages
Lustre: mdt01_014: service thread pid 27329 was inactive for 40.109 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 14085, comm: mdt04_010 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf7/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22113, comm: mdt07_017 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf7/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt07_012: service thread pid 27669 was inactive for 40.135 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Pid: 27329, comm: mdt01_014 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_object_lock+0x88/0x1c0 [mdt]
[<0>] mdt_getattr_name_lock+0xf7/0x2bd0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1c9/0xc70 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3b1/0xbb0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x35b/0x1800 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 24087:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff8800711e6fc8: cannot apply new layout on [0x240000403:0xe58:0x0] : rc = -5
LustreError: 24087:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000403:0xe58:0x0] error -5.
Lustre: dir [0x200000404:0x13b6:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 7 previous similar messages
LustreError: 25012:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000403:0xe1a:0x0]: rc = -5
LustreError: 25012:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 48 previous similar messages
LustreError: 25012:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 25012:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 48 previous similar messages
Lustre: mdt06_000: service thread pid 32321 was inactive for 40.064 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000403:0xe2c:0x0]/0xa): rc = 0
Lustre: Skipped 1 previous similar message
LustreError: 3847:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff8800711e6fc8: cannot apply new layout on [0x240000403:0xe58:0x0] : rc = -5
Lustre: mdt00_017: service thread pid 20366 was inactive for 74.185 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 31364:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0xda1:0x0] migrate mdt count mismatch 2 != 3
Lustre: mdt02_009: service thread pid 14099 was inactive for 74.161 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 32337:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0xf42:0x0]: rc = -2
LustreError: 32329:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0xf42:0x0]: rc = -2
LustreError: 32270:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff880252dd25c0/0xb29c9b2cac722318 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0xfab:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb29c9b2cac7222fc expref: 339 pid: 27329 timeout: 30239 lvb_type: 0
LustreError: 26300:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff88024e428958 ns: mdt-lustre-MDT0001_UUID lock: ffff88024220c3c0/0xb29c9b2cac7228f9 lrc: 3/0,0 mode: PR/PR res: [0x240000402:0x1d:0x0].0x0 bits 0x12/0x0 rrc: 50 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb29c9b2cac7228eb expref: 70 pid: 26300 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff8800711e6fc8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff8800711e6fc8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff8800711e6fc8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 2606:0:(file.c:5550:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0xcf4:0x0] error: rc = -5
LustreError: 2606:0:(file.c:5550:ll_inode_revalidate_fini()) Skipped 6 previous similar messages
LustreError: 7857:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000404:0xfab:0x0] error -5.
Lustre: lustre-MDT0001-mdc-ffff8800711e6fc8: Connection restored to (at 0@lo)
LustreError: 21013:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) ### lock on destroyed export ffff88024e428958 ns: mdt-lustre-MDT0001_UUID lock: ffff8802c4487840/0xb29c9b2cac7ed381 lrc: 3/0,0 mode: CR/CR res: [0x240000404:0x1299:0x0].0x0 bits 0xa/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xb29c9b2cac7ed373 expref: 8 pid: 21013 timeout: 0 lvb_type: 0
LustreError: 21013:0:(ldlm_lockd.c:1499:ldlm_handle_enqueue()) Skipped 22 previous similar messages
LustreError: 21891:0:(mdd_object.c:3829:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x113b:0x0]: rc = -2
LustreError: 23588:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 11 [0x280000403:0xe1a:0x0]: rc = -5
LustreError: 23588:0:(statahead.c:792:ll_statahead_interpret_work()) Skipped 3 previous similar messages
LustreError: 24094:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x280000404:0xc9f:0x0] : rc = -5
LustreError: 24094:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000404:0x127c:0x0]/0xa): rc = 0
LustreError: 1542:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 20 [0x280000404:0x1690:0x0]: rc = -5
LustreError: 9792:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000403:0x1560:0x0] migrate mdt count mismatch 3 != 2
LustreError: 12765:0:(statahead.c:792:ll_statahead_interpret_work()) lustre: getattr callback for 20 [0x280000404:0x1690:0x0]: rc = -5
LustreError: 27318:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0x10f6:0x0] migrate mdt count mismatch 3 != 2
Lustre: dir [0x200000403:0x15c2:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 11 previous similar messages
LustreError: 32379:0:(mdt_reint.c:2451:mdt_reint_migrate()) lustre-MDT0000: migrate [0x200000403:0x2:0x0]/17 failed: rc = -114
LustreError: 32379:0:(mdt_reint.c:2451:mdt_reint_migrate()) Skipped 357 previous similar messages
9[27570]: segfault at 8 ip 00007fa4d51697e8 sp 00007ffc313daef0 error 4 in ld-2.17.so[7fa4d515e000+22000]
8[4515]: segfault at 8 ip 00007fbe113557e8 sp 00007ffef1dc8ae0 error 4 in ld-2.17.so[7fbe1134a000+22000]
LustreError: 5440:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8800711e6fc8: inode [0x240000405:0x715:0x0] mdc close failed: rc = -116
LustreError: 5440:0:(file.c:264:ll_close_inode_openhandle()) Skipped 75 previous similar messages
7[7042]: segfault at 8 ip 00007f7a38b107e8 sp 00007ffc2363f230 error 4 in ld-2.17.so[7f7a38b05000+22000]
LustreError: 20518:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x240000405:0x81f:0x0] : rc = -5
LustreError: 20518:0:(vvp_io.c:1923:vvp_io_init()) lustre: refresh file layout [0x240000405:0x81f:0x0] error -5.
1[26374]: segfault at 8 ip 00007f8a675857e8 sp 00007fff7edb6200 error 4 in ld-2.17.so[7f8a6757a000+22000]
LustreError: 24628:0:(mdt_xattr.c:415:mdt_dir_layout_update()) lustre-MDT0000: [0x200000404:0x2027:0x0] migrate mdt count mismatch 3 != 2
Lustre: 30911:0:(lod_lov.c:1433:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x18dd:0x0] with magic=0xbd60bd0
Lustre: 30911:0:(lod_lov.c:1433:lod_parse_striping()) Skipped 133 previous similar messages
LustreError: 4169:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) lustre-MDD0002: '7' migration was interrupted, run 'lfs migrate -m 0 -c 3 -H crush 7' to finish migration: rc = -1
LustreError: 4169:0:(mdd_dir.c:4470:mdd_migrate_cmd_check()) Skipped 42 previous similar messages
LustreError: 7992:0:(lov_object.c:1360:lov_layout_change()) lustre-clilov-ffff880264586678: cannot apply new layout on [0x240000405:0x81f:0x0] : rc = -5
LustreError: 7992:0:(lov_object.c:1360:lov_layout_change()) Skipped 1 previous similar message
LustreError: 7992:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000405:0x81f:0x0]: rc = -5
LustreError: 7992:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 131 previous similar messages
LustreError: 7992:0:(llite_lib.c:3691:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 7992:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 131 previous similar messages
ptlrpc_watchdog_fire: 24 callbacks suppressed
Lustre: mdt_io07_002: service thread pid 32388 was inactive for 40.145 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 32388, comm: mdt_io07_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_pdo_lock+0x4d9/0x7e0 [mdt]
[<0>] mdt_parent_lock+0x76/0x2a0 [mdt]
[<0>] mdt_reint_migrate+0xe24/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 32366, comm: mdt_io00_001 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x151/0x2f0 [osp]
[<0>] lod_object_lock+0xdb/0x7c0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x14c/0x189 [mdt]
[<0>] mdt_object_lock_internal+0x377/0x420 [mdt]
[<0>] mdt_rename_lock+0xc3/0x2d0 [mdt]
[<0>] mdt_reint_migrate+0x891/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 7451, comm: mdt_io04_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x963/0xd00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1a9/0x420 [mdt]
[<0>] mdt_rename_lock+0xc3/0x2d0 [mdt]
[<0>] mdt_reint_migrate+0x891/0x2420 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x74c/0xbb0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x74e/0x1a50 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x26c/0xcb0 [ptlrpc]
[<0>] ptlrpc_main+0xc66/0x1670 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt_io02_003: service thread pid 16547 was inactive for 40.100 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 32270:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8801513b7840/0xb29c9b2cacaa77f1 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x1f70:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xb29c9b2cacaa77c0 expref: 222 pid: 3521 timeout: 30841 lvb_type: 0
LustreError: 11-0: lustre-MDT0000-mdc-ffff8800711e6fc8: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 21 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff8800711e6fc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0000-mdc-ffff8800711e6fc8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 18031:0:(llite_lib.c:2019:ll_md_setattr()) md_setattr fails: rc = -108
Lustre: lustre-MDT0000-mdc-ffff8800711e6fc8: Connection restored to (at 0@lo)
1[25185]: segfault at 8 ip 00007f0c8edd37e8 sp 00007ffc19a6ff90 error 4 in ld-2.17.so[7f0c8edc8000+22000]
LustreError: 22554:0:(mdt_handler.c:777:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0x1808:0x0] ACL: rc = -2
5[1909]: segfault at 8 ip 00007f10522077e8 sp 00007ffcad2859e0 error 4 in ld-2.17.so[7f10521fc000+22000]

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt04_000:22473]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm floppy serio_raw drm_panel_orientation_quirks libata virtio_blk i2c_core [last unloaded: libcfs]
CPU: 9 PID: 22473 Comm: mdt04_000 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880328d649d0 ti: ffff880289254000 task.ti: ffff880289254000
RIP: 0010:[<ffffffffa0cc9766>] [<ffffffffa0cc9766>] zap_leaf_lookup+0xb6/0x160 [zfs]
RSP: 0018:ffff880289257430 EFLAGS: 00000213
RAX: 0000000000000000 RBX: ffff8802892573c0 RCX: 0000000000000009
RDX: 000000000000000e RSI: 0000000000000009 RDI: fe87807a7a290000
RBP: ffff880289257460 R08: 0000000000000000 R09: ffff88031cbd9000
R10: 0000000000000000 R11: ffff8802cb90f208 R12: 0000000000000000
R13: ffff880289257498 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000040873d CR3: 000000006dc5a000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0cc755c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c2aacd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0cc7651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0cccdfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0ccba97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0ccd085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa119a017>] osd_dir_insert+0x277/0x990 [osd_zfs]
[<ffffffffa139a04a>] lod_sub_insert+0x1ca/0x3b0 [lod]
[<ffffffffa1369e76>] lod_insert+0x16/0x20 [lod]
[<ffffffffa11e92af>] __mdd_index_insert_only+0xbf/0x2d0 [mdd]
[<ffffffffa11eda42>] ? mdd_create_object+0xb2/0xa60 [mdd]
[<ffffffffa11e9509>] __mdd_index_insert+0x49/0x250 [mdd]
[<ffffffffa11f472e>] mdd_create+0x124e/0x1d30 [mdd]
[<ffffffffa129cc00>] mdt_reint_open+0x2ad0/0x2c10 [mdt]
[<ffffffffa03e1c3e>] ? upcall_cache_get_entry+0x5ae/0x930 [obdclass]
[<ffffffffa128c617>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa126157c>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<ffffffffa126dae3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa12656ac>] mdt_intent_opc+0x1dc/0xc40 [mdt]
[<ffffffffa126da50>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa126b63a>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa06658b1>] ldlm_lock_enqueue+0x3e1/0xbe0 [ptlrpc]
[<ffffffffa01de175>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa01e13e6>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa068ef2f>] ldlm_handle_enqueue+0x8bf/0x17e0 [ptlrpc]
[<ffffffffa060ce00>] ? lustre_msg_buf_v2+0x130/0x1f0 [ptlrpc]
[<ffffffffa06cf728>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa06d41ba>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<ffffffffa061fe21>] ptlrpc_server_handle_request+0x251/0xc00 [ptlrpc]
[<ffffffffa0621af1>] ptlrpc_main+0xc21/0x15f0 [ptlrpc]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0620ed0>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt04_002:13019]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 9 PID: 13019 Comm: mdt04_002 Kdump: loaded Tainted: P B OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802ccba1280 ti: ffff8802f4384000 task.ti: ffff8802f4384000
RIP: 0010:[<ffffffffa0cdd75a>] [<ffffffffa0cdd75a>] zap_leaf_lookup+0xaa/0x160 [zfs]
RSP: 0018:ffff8802f4387410 EFLAGS: 00000213
RAX: 0000000000000000 RBX: ffff8802f43873a0 RCX: ffff88008b33d648
RDX: 000000000000000e RSI: 0000000000000200 RDI: 67939a94bc6f0000
RBP: ffff8802f4387440 R08: 0000000000000000 R09: ffff8802cf8b9000
R10: 0000000000000000 R11: ffff880331404cc0 R12: 0000000000000000
R13: ffff8802f4387478 R14: 0000000000000001 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001f36d7c CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa0cdb55c>] fzap_add_cd+0xdc/0x180 [zfs]
[<ffffffffa0c3eacd>] ? dmu_object_info_from_dnode+0x7d/0xb0 [zfs]
[<ffffffffa0cdb651>] fzap_add+0x51/0x60 [zfs]
[<ffffffffa0ce0dfd>] zap_add_impl+0xfd/0x260 [zfs]
[<ffffffffa0cdfa97>] ? zap_lockdir_by_dnode.constprop.14+0x97/0xc0 [zfs]
[<ffffffffa0ce1085>] zap_add_by_dnode+0x75/0xa0 [zfs]
[<ffffffffa0a1f017>] osd_dir_insert+0x277/0x990 [osd_zfs]
[<ffffffffa13a9c4e>] lod_sub_insert+0x1be/0x3a0 [lod]
[<ffffffffa1379b16>] lod_insert+0x16/0x20 [lod]
[<ffffffffa11f8b5f>] __mdd_index_insert_only+0xbf/0x2d0 [mdd]
[<ffffffffa11fd312>] ? mdd_create_object+0xb2/0xa60 [mdd]
[<ffffffffa11f8db9>] __mdd_index_insert+0x49/0x250 [mdd]
[<ffffffffa1204166>] mdd_create+0x1246/0x1d30 [mdd]
[<ffffffffa12acfbf>] mdt_reint_open+0x2ebf/0x3000 [mdt]
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffffa036ed7e>] ? upcall_cache_get_entry+0x5ae/0x930 [obdclass]
[<ffffffffa128700d>] ? ucred_set_rbac_roles+0x2d/0x100 [mdt]
[<ffffffffa129c797>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa127239c>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<ffffffffa127e7a3>] mdt_intent_open+0x93/0x480 [mdt]
[<ffffffffa127643c>] mdt_intent_opc+0x1dc/0xc40 [mdt]
[<ffffffffa127e710>] ? mdt_intent_fixup_resent+0x240/0x240 [mdt]
[<ffffffffa127c2fa>] mdt_intent_policy+0xfa/0x460 [mdt]
[<ffffffffa06a4b51>] ldlm_lock_enqueue+0x3e1/0xbe0 [ptlrpc]
[<ffffffffa01c02c5>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa01c3536>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa06cd40f>] ldlm_handle_enqueue+0x8bf/0x17c0 [ptlrpc]
[<ffffffffa064cf00>] ? lustre_msg_buf_v2+0x140/0x1e0 [ptlrpc]
[<ffffffffa070eae8>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<ffffffffa071904a>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<ffffffffa0660108>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<ffffffffa0661dd1>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa06611a0>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: 13010:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e1b6db40 x1763280405029888/t4294968056(0) o101->d00bdb4b-4192-4079-8a3c-4cfe2ffcbf78@0@lo:510/0 lens 376/864 e 0 to 0 dl 1681595400 ref 1 fl Interpret:H/202/0 rc 0/0 uid:0 gid:0 job:'cat.0'
LustreError: 28230:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x240000404:0x4e:0x0] mdc close failed: rc = -116
LustreError: 29127:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x240000404:0x7:0x0] mdc close failed: rc = -116
LustreError: 29127:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 30259:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x200000404:0x3b:0x0] mdc close failed: rc = -116
LustreError: 29110:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d06a548: inode [0x200000404:0x3c:0x0] mdc close failed: rc = -2
LustreError: 32604:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x280000404:0x75:0x0] mdc close failed: rc = -2
LustreError: 32604:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x280000403:0x10e:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 29734:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '5' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 5' to finish migration.
LustreError: 14078:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x200000404:0x8f:0x0] mdc close failed: rc = -2
Lustre: dir [0x280000403:0x10b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 11 previous similar messages
LustreError: 23507:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '19' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 19' to finish migration.
Lustre: 22881:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x288:0x0] with magic=0xbd60bd0
Lustre: 3604:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff88024892eec0 x1763280409588672/t4294971233(0) o101->d00bdb4b-4192-4079-8a3c-4cfe2ffcbf78@0@lo:619/0 lens 376/864 e 0 to 0 dl 1681595509 ref 1 fl Interpret:H/202/0 rc 0/0 uid:0 gid:0 job:'dd.0'
LustreError: 30732:0:(mdt_handler.c:775:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x1d9:0x0] ACL: rc = -2
Lustre: lustre-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x200000403:0xf3:0x0]/0xa): rc = 0
Lustre: 13006:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x2f1:0x0] with magic=0xbd60bd0
Lustre: 13006:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 32333:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x280000404:0x139:0x0] mdc close failed: rc = -2
LustreError: 32333:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 11035:0:(mdt_recovery.c:150:mdt_req_from_lrd()) @@@ restoring transno req@ffff8801aeaaeec0 x1763280411669120/t4294975825(0) o101->d00bdb4b-4192-4079-8a3c-4cfe2ffcbf78@0@lo:668/0 lens 376/816 e 0 to 0 dl 1681595558 ref 1 fl Interpret:H/202/0 rc 0/0 uid:0 gid:0 job:'dd.0'
Lustre: 1117:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0xde:0x0] with magic=0xbd60bd0
Lustre: 1117:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 15 previous similar messages
Lustre: 11024:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x165:0x0] with magic=0xbd60bd0
Lustre: 11024:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 23957:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1b0:0x0] with magic=0xbd60bd0
Lustre: 23957:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 1 previous similar message
Lustre: dir [0x200000403:0x1b6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 368:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '18' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 18' to finish migration.
LustreError: 25470:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '8' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 8' to finish migration.
LustreError: 21905:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x200000403:0x1fc:0x0] mdc close failed: rc = -116
LustreError: 9568:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d06a548: inode [0x280000403:0x59f:0x0] mdc close failed: rc = -116
LustreError: 9568:0:(file.c:242:ll_close_inode_openhandle()) Skipped 3 previous similar messages
11[17481]: segfault at 8 ip 00007f647c0867e8 sp 00007ffc85709de0 error 4 in ld-2.17.so[7f647c07b000+22000]
LustreError: 3045:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '10' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 10' to finish migration.
LustreError: 32414:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x245:0x0]: rc = -2
Lustre: 30732:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x5f6:0x0] with magic=0xbd60bd0
Lustre: 30732:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 3 previous similar messages
7[5651]: segfault at 8 ip 00007f7157e107e8 sp 00007fffc7af8b60 error 4 in ld-2.17.so[7f7157e05000+22000]
7[6043]: segfault at 8 ip 00007f6f9db137e8 sp 00007ffd4956b900 error 4 in ld-2.17.so[7f6f9db08000+22000]
Lustre: 21965:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x761:0x0] with magic=0xbd60bd0
Lustre: 21965:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 5 previous similar messages
Lustre: dir [0x240000404:0x6ee:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 28255:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '13' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 13' to finish migration.
LustreError: 30731:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '14' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 14' to finish migration.
LustreError: 30731:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 1 previous similar message
11[25486]: segfault at 0 ip (null) sp 00007ffd4a2b70a8 error 14 in 11[400000+6000]
Lustre: 23507:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000404:0x886:0x0] with magic=0xbd60bd0
Lustre: 23507:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 7 previous similar messages
Lustre: mdt07_007: service thread pid 24004 was inactive for 62.051 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 24004, comm: mdt07_007 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_object_lock_internal+0x43e/0x5d0 [mdt]
[<0>] mdt_object_lock+0x95/0x1d0 [mdt]
[<0>] mdt_reint_migrate+0x1b5f/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 13013, comm: mdt02_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt05_003: service thread pid 22538 was inactive for 62.116 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 22538, comm: mdt05_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt03_013: service thread pid 30755 was inactive for 62.152 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt02_004: service thread pid 22710 was inactive for 62.171 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt02_014: service thread pid 368 was inactive for 62.062 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: mdt01_002: service thread pid 13010 was inactive for 66.111 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
Lustre: mdt04_000: service thread pid 13017 was inactive for 62.163 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
LustreError: 12906:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802507cc3c0/0xf75f0e3dcfeec064 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x349:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf75f0e3dcfeec041 expref: 200 pid: 25470 timeout: 335581 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff8802951db7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff8802951db7e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff8802951db7e8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 28170:0:(llite_lib.c:1971:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 27498:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x240000404:0x35c:0x0] mdc close failed: rc = -108
LustreError: 27498:0:(file.c:242:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff8802951db7e8: Connection restored to 192.168.123.112@tcp (at 0@lo)
Lustre: dir [0x200000403:0x53d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 15 previous similar messages
LustreError: 26343:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration.
Lustre: lustre-MDT0001: trigger partial OI scrub for RPC inconsistency, checking FID [0x240000403:0x848:0x0]/0xa): rc = 0
Lustre: 30695:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x5d3:0x0] with magic=0xbd60bd0
Lustre: 30695:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 9 previous similar messages
LustreError: 22001:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '9' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 9' to finish migration.
LustreError: 22001:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
19[23056]: segfault at 8 ip 00007fbea04b37e8 sp 00007fff0ec8d000 error 4 in ld-2.17.so[7fbea04a8000+22000]
2[10832]: segfault at 8 ip 00007f92450257e8 sp 00007ffd5ac32740 error 4 in ld-2.17.so[7f924501a000+22000]
6[11128]: segfault at 8 ip 00007f0db201d7e8 sp 00007ffd0c8c06a0 error 4 in ld-2.17.so[7f0db2012000+22000]
Lustre: dir [0x200000403:0x79d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 3412:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0xc78:0x0] with magic=0xbd60bd0
Lustre: 3412:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 19 previous similar messages
LustreError: 25509:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '4' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 4' to finish migration.
LustreError: 25509:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 6 previous similar messages
Lustre: dir [0x240000405:0x3c6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 6537:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000405:0x3c6:0x0]: rc = -2
LustreError: 6537:0:(mdd_object.c:3491:mdd_close()) Skipped 1 previous similar message
LustreError: 6710:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88028d06a548: inode [0x240000405:0x3c6:0x0] mdc close failed: rc = -2
LustreError: 6710:0:(file.c:242:ll_close_inode_openhandle()) Skipped 14 previous similar messages
LustreError: 32554:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000405:0x3c6:0x0]: rc = -2
18[8639]: segfault at 8 ip 00007f62430e27e8 sp 00007ffd4040b3a0 error 4 in ld-2.17.so[7f62430d7000+22000]
1[11692]: segfault at 8 ip 00007fc0556d77e8 sp 00007ffe67d48e50 error 4 in ld-2.17.so[7fc0556cc000+22000]
Lustre: dir [0x200000403:0xaa8:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
6[17800]: segfault at 8 ip 00007f0b414ae7e8 sp 00007ffc8ed9fbb0 error 4 in ld-2.17.so[7f0b414a3000+22000]
ptlrpc_watchdog_fire: 19 callbacks suppressed
Lustre: mdt06_006: service thread pid 26343 was inactive for 86.307 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 26343, comm: mdt06_006 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_object_lock+0x95/0x1d0 [mdt]
[<0>] mdt_getattr_name_lock+0xbb9/0x29c0 [mdt]
[<0>] mdt_intent_getattr+0x2c5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dc/0xc40 [mdt]
[<0>] mdt_intent_policy+0xfa/0x460 [mdt]
[<0>] ldlm_lock_enqueue+0x3e1/0xbe0 [ptlrpc]
[<0>] ldlm_handle_enqueue+0x8bf/0x17c0 [ptlrpc]
[<0>] tgt_enqueue+0x68/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 12906:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802c6c61e40/0xf75f0e3dd00bacea lrc: 3/0,0 mode: PR/PR res: [0x280000404:0xf1b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf75f0e3dd00bacc7 expref: 295 pid: 30755 timeout: 336004 lvb_type: 0
LustreError: 11-0: lustre-MDT0002-mdc-ffff8802951db7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0002-mdc-ffff8802951db7e8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0002-mdc-ffff8802951db7e8: This client was evicted by lustre-MDT0002; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
LustreError: 31150:0:(llite_lib.c:1971:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 31964:0:(llite_lib.c:3408:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 31159:0:(file.c:5324:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -108
LustreError: 29531:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0002-mdc-ffff8802951db7e8: namespace resource [0x280000404:0x10d3:0x0].0x0 (ffff8801ca828040) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31268:0:(mdc_request.c:1484:mdc_read_page()) lustre-MDT0002-mdc-ffff8802951db7e8: [0x280000403:0x1:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0002-mdc-ffff8802951db7e8: Connection restored to 192.168.123.112@tcp (at 0@lo)
LustreError: 30755:0:(mdt_open.c:1231:mdt_cross_open()) lustre-MDT0001: [0x240000405:0x8ec:0x0] doesn't exist!: rc = -14
Lustre: dir [0x240000403:0x12ae:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
2[6897]: segfault at 8 ip 00007f6e063a87e8 sp 00007ffc0031e380 error 4 in ld-2.17.so[7f6e0639d000+22000]
Lustre: 29911:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000405:0xa13:0x0] with magic=0xbd60bd0
Lustre: 29911:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 77 previous similar messages
LustreError: 13014:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0001: '3' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 3' to finish migration.
LustreError: 13014:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 11 previous similar messages
2[20865]: segfault at 8 ip 00007f4f455847e8 sp 00007fff349d8fe0 error 4 in ld-2.17.so[7f4f45579000+22000]
2[27056]: segfault at 8 ip 00007efee15007e8 sp 00007ffe7b6ca7e0 error 4 in ld-2.17.so[7efee14f5000+22000]
LustreError: 23888:0:(mdt_handler.c:775:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000405:0xbc6:0x0] ACL: rc = -2
9[31595]: segfault at 4045bc ip 00000000004045bc sp 00007ffff7e40088 error 7 in 9[400000+6000]
Lustre: mdt04_016: service thread pid 14998 was inactive for 40.065 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 14998, comm: mdt04_016 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_rename+0x143d/0x28e0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 3045, comm: mdt02_017 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_object_lock+0x95/0x1d0 [mdt]
[<0>] mdt_object_stripes_lock+0x146/0x650 [mdt]
[<0>] mdt_reint_migrate+0xa20/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt00_011: service thread pid 30723 was inactive for 40.031 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt02_008: service thread pid 29466 was inactive for 40.042 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt04_011: service thread pid 1117 was inactive for 40.122 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 12906:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802c81abc40/0xf75f0e3dd02467f3 lrc: 3/0,0 mode: PR/PR res: [0x240000405:0xb4d:0x0].0x0 bits 0x12/0x0 rrc: 14 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf75f0e3dd02467e5 expref: 345 pid: 21172 timeout: 336328 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff88028d06a548: operation mds_close to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff88028d06a548: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff88028d06a548: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 3197:0:(llite_lib.c:3408:ll_prep_inode()) new_inode -fatal: rc -2
Lustre: lustre-MDT0001-mdc-ffff88028d06a548: Connection restored to 192.168.123.112@tcp (at 0@lo)
LustreError: 20205:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x240000405:0xf07:0x0] mdc close failed: rc = -2
LustreError: 20205:0:(file.c:242:ll_close_inode_openhandle()) Skipped 54 previous similar messages
Lustre: dir [0x240000405:0xefe:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 35 previous similar messages
LustreError: 23859:0:(mdd_object.c:3491:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x1cad:0x0]: rc = -2
LustreError: 13012:0:(mdt_handler.c:775:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000406:0x153:0x0] ACL: rc = -2
19[30464]: segfault at 8 ip 00007f22e1d007e8 sp 00007ffe87101510 error 4 in ld-2.17.so[7f22e1cf5000+22000]
5[1321]: segfault at 8 ip 00007ffabd3937e8 sp 00007ffc9dd4b030 error 4 in ld-2.17.so[7ffabd388000+22000]
2[1766]: segfault at 0 ip (null) sp 00007fffb100a128 error 14 in 2[400000+6000]
5[1430]: segfault at 8 ip 00007f00975567e8 sp 00007ffd6393cee0 error 4 in ld-2.17.so[7f009754b000+22000]
7[12355]: segfault at 8 ip 00007fbc7cb807e8 sp 00007ffc7c151860 error 4 in ld-2.17.so[7fbc7cb75000+22000]
LustreError: 13013:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0000: '4' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 4' to finish migration.
LustreError: 13013:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 17 previous similar messages
LustreError: 22881:0:(mdt_open.c:1231:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x129b:0x0] doesn't exist!: rc = -14
Lustre: 23411:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0xab7:0x0] with magic=0xbd60bd0
Lustre: 23411:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 75 previous similar messages
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000403:0x1f8b:0x0]/0xa): rc = 0
Lustre: dir [0x280000403:0x1f8b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 41 previous similar messages
6[27410]: segfault at 8 ip 00007fa9a7f467e8 sp 00007fff3fe07050 error 4 in ld-2.17.so[7fa9a7f3b000+22000]
ptlrpc_watchdog_fire: 14 callbacks suppressed
Lustre: ll_ost06_003: service thread pid 20531 was inactive for 64.132 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 20531, comm: ll_ost06_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x385/0x750 [ofd]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 12906:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8802b8b60040/0xf75f0e3dd0424073 lrc: 3/0,0 mode: PW/PW res: [0x380000402:0x258:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1441792->1507327) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xf75f0e3dd042406c expref: 31 pid: 20290 timeout: 336881 lvb_type: 0
LustreError: 11-0: lustre-OST0003-osc-ffff88028d06a548: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff88028d06a548: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 7334:0:(osc_request.c:1037:osc_init_grant()) lustre-OST0003-osc-ffff88028d06a548: granted 3407872 but already consumed 8519680
LustreError: 167-0: lustre-OST0003-osc-ffff88028d06a548: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 7343:0:(llite_lib.c:3802:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000405:0x18aa:0x0]// may get corrupted (rc -108)
Lustre: 7343:0:(llite_lib.c:3802:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x2524:0x0]// may get corrupted (rc -108)
Lustre: 7343:0:(llite_lib.c:3802:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000405:0x189a:0x0]/ may get corrupted (rc -108)
Lustre: 7343:0:(llite_lib.c:3802:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000406:0x752:0x0]/ may get corrupted (rc -108)
LustreError: 26463:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-OST0003-osc-ffff88028d06a548: namespace resource [0x380000402:0x258:0x0].0x0 (ffff8800796dafc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 26463:0:(ldlm_resource.c:1125:ldlm_resource_complain()) Skipped 3 previous similar messages
Lustre: lustre-OST0003-osc-ffff88028d06a548: Connection restored to 192.168.123.112@tcp (at 0@lo)
LustreError: 13019:0:(mdd_object.c:403:mdd_xattr_get()) lustre-MDD0002: object [0x280000405:0xe84:0x0] not found: rc = -2
12[14373]: segfault at 0 ip (null) sp 00007ffec8a57238 error 14 in 12[400000+6000]
3[30710]: segfault at 0 ip (null) sp 00007ffd94e9b898 error 14 in 3[400000+6000]
Lustre: mdt07_002: service thread pid 13028 was inactive for 40.087 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 13028, comm: mdt07_002 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_object_lock+0x95/0x1d0 [mdt]
[<0>] mdt_object_stripes_lock+0x146/0x650 [mdt]
[<0>] mdt_reint_migrate+0xa20/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22629, comm: mdt07_003 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_lock_internal+0x1ba/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt07_013: service thread pid 29627 was inactive for 40.067 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 5 previous similar messages
Lustre: mdt03_014: service thread pid 30779 was inactive for 44.131 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt03_002: service thread pid 13016 was inactive for 44.133 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt03_005: service thread pid 23666 was inactive for 44.025 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt04_002: service thread pid 13019 was inactive for 94.234 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
LustreError: 12906:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802898bb4c0/0xf75f0e3dd051c093 lrc: 3/0,0 mode: PR/PR res: [0x240000406:0x62d:0x0].0x0 bits 0x12/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xf75f0e3dd051c085 expref: 357 pid: 29911 timeout: 337082 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff8802951db7e8: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0001-mdc-ffff8802951db7e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff8802951db7e8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 26141:0:(file.c:5324:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000406:0x73f:0x0] error: rc = -108
LustreError: 14705:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802951db7e8: inode [0x240000405:0x18ce:0x0] mdc close failed: rc = -5
LustreError: 14705:0:(file.c:242:ll_close_inode_openhandle()) Skipped 18 previous similar messages
LustreError: 22215:0:(llite_lib.c:3408:ll_prep_inode()) new_inode -fatal: rc -2
Lustre: lustre-MDT0001-mdc-ffff8802951db7e8: Connection restored to 192.168.123.112@tcp (at 0@lo)
8[7425]: segfault at 8 ip 00007f3867e377e8 sp 00007ffed2c71770 error 4 in ld-2.17.so[7f3867e2c000+22000]
18[19141]: segfault at 8 ip 00007f50b74537e8 sp 00007ffcb7c17410 error 4 in ld-2.17.so[7f50b7448000+22000]
Lustre: dir [0x200000403:0x1a11:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 74 previous similar messages
LustreError: 662:0:(lustre_lmv.h:452:lmv_is_sane()) unknown layout LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate_offset=1 migrate_hash=fnv_1a_64:2 pool=
Lustre: 13022:0:(lod_lov.c:1326:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000406:0xe77:0x0] with magic=0xbd60bd0
Lustre: 13022:0:(lod_lov.c:1326:lod_parse_striping()) Skipped 31 previous similar messages
LustreError: 6751:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) lustre-MDD0002: '17' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 17' to finish migration.
LustreError: 6751:0:(mdd_dir.c:4340:mdd_migrate_cmd_check()) Skipped 29 previous similar messages
LustreError: 30824:0:(mdt_open.c:1231:mdt_cross_open()) lustre-MDT0000: [0x200000403:0x19d8:0x0] doesn't exist!: rc = -14
14[11030]: segfault at 0 ip (null) sp 00007ffec26d3bb8 error 14 in 14[400000+6000]
ptlrpc_watchdog_fire: 11 callbacks suppressed
Lustre: mdt01_013: service thread pid 11035 was inactive for 44.086 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 11035, comm: mdt01_013 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_object_lock_internal+0x43e/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 7997, comm: mdt00_014 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_object_lock_internal+0x43e/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22881, comm: mdt01_004 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] ldlm_completion_ast+0x857/0xab0 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa24/0xee0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x461/0xa70 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_object_lock_internal+0x43e/0x5d0 [mdt]
[<0>] mdt_rename_lock+0xcb/0x2e0 [mdt]
[<0>] mdt_reint_migrate+0x7be/0x1c00 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xba0 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x288/0xc20 [ptlrpc]
[<0>] ptlrpc_main+0xc31/0x1600 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt01_010: service thread pid 30841 was inactive for 40.080 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages