BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fc11c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 27cb02067 PUD 292c13067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy libata i2c_core [last unloaded: obdecho]
CPU: 0 PID: 27808 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880239fe49d0 ti: ffff8802d6208000 task.ti: ffff8802d6208000
RIP: 0010:[<ffffffffa15fc11c>] [<ffffffffa15fc11c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8802d620bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88013623e6b8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802b6addfc8 RDI: ffff8802b6addfa8
RBP: ffff8802d620bc48 R08: ffff880276e316f8 R09: ffff8802d620ba68
R10: ffff88013623ae68 R11: ffff8801362389f8 R12: ffff88018074e718
R13: 0000000000000000 R14: ffff8801e1fd0a58 R15: ffff880089bce738
FS: 00007f0c68299740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000320b36000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa15e8bef>] ll_dir_ioctl+0x5d0f/0x6fe0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 27792:0:(llite_lib.c:1937:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 27792:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88031dd2efc8: inode [0x2000059f3:0x92f0:0x0] mdc close failed: rc = -28
LustreError: 27792:0:(file.c:242:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 27808:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 27808:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1705e65>] ll_migrate+0xa45/0xff0 [lustre]
PGD 1c203e067 PUD 1d5228067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: obdecho]
CPU: 3 PID: 23150 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880127b4a4f0 ti: ffff8801a26f4000 task.ti: ffff8801a26f4000
RIP: 0010:[<ffffffffa1705e65>] [<ffffffffa1705e65>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff8801a26f7bc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801046c9e48 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8801d1a10ad8
RBP: ffff8801a26f7c40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff8800a2d84968
R13: 0000000000000000 R14: ffff8800a5350d48 R15: ffff8801046ce288
FS: 00007fc05791d740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000204426000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa16eeff7>] ll_dir_ioctl+0x6bc7/0x7860 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810a2009>] ? mod_timer+0x119/0x260
[<ffffffff810a2168>] ? add_timer+0x18/0x20
[<ffffffff810b10ab>] ? __queue_delayed_work+0x8b/0x1a0
[<ffffffff817e32dc>] ? _raw_spin_unlock_bh+0x1c/0x20
[<ffffffff811d92ef>] ? bdi_wakeup_thread_delayed+0x5f/0x70
[<ffffffff81277f01>] ? __mark_inode_dirty+0x161/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
LustreError: 799:0:(osd_handler.c:207:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 799:0:(osd_handler.c:207:osd_trans_start()) Skipped 4 previous similar messages
LustreError: 23041:0:(llite_lib.c:1996:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 23041:0:(file.c:247:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cfa4e678: inode [0x240002b15:0x94fd:0x0] mdc close failed: rc = -28
Lustre: DEBUG MARKER: hardlinks
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1690315>] ll_migrate+0xa45/0xff0 [lustre]
PGD 2f555c067 PUD 308aff067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw floppy libata virtio_blk i2c_core [last unloaded: obdecho]
CPU: 13 PID: 27986 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880301621280 ti: ffff8802e8c34000 task.ti: ffff8802e8c34000
RIP: 0010:[<ffffffffa1690315>] [<ffffffffa1690315>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff8802e8c37bc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880249ffc548 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880077da8fd8
RBP: ffff8802e8c37c40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff8802846e5138
R13: 0000000000000000 R14: ffff8802464c1738 R15: ffff880249ffcf08
FS: 00007f95c305e740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000308938000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa167ada4>] ll_dir_ioctl+0x7184/0x7ed0 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa022fc17>] ? cfs_trace_unlock_tcd+0x27/0x70 [libcfs]
[<ffffffffa02304a4>] ? libcfs_debug_msg+0x6d4/0xc20 [libcfs]
[<ffffffffa03b5be9>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81277f48>] ? __mark_inode_dirty+0x1a8/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
Lustre: DEBUG MARKER: hardlinks
LustreError: 14261:0:(osd_handler.c:207:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 14261:0:(osd_handler.c:207:osd_trans_start()) Skipped 2 previous similar messages
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1697315>] ll_migrate+0xa45/0xff0 [lustre]
PGD 28b850067 PUD 175548067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix libata floppy virtio_blk serio_raw i2c_core [last unloaded: obdecho]
CPU: 3 PID: 22885 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88028d5b3760 ti: ffff88024565c000 task.ti: ffff88024565c000
RIP: 0010:[<ffffffffa1697315>] [<ffffffffa1697315>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff88024565fbc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880002eecf08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8801bfd35558
RBP: ffff88024565fc40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff8801273c1c28
R13: 0000000000000000 R14: ffff88022df4e6b8 R15: ffff880002eed8c8
FS: 00007fbb23449740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000018b1d0000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1681da4>] ll_dir_ioctl+0x7184/0x7ed0 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0255c17>] ? cfs_trace_unlock_tcd+0x27/0x70 [libcfs]
[<ffffffffa02564a4>] ? libcfs_debug_msg+0x6d4/0xc20 [libcfs]
[<ffffffffa03b5be9>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81277f48>] ? __mark_inode_dirty+0x1a8/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
Lustre: DEBUG MARKER: hardlinks
LustreError: 2645:0:(osd_handler.c:207:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa16b1fb5>] ll_migrate+0xa45/0xff0 [lustre]
PGD 1fb40067 PUD 1fb41067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: obdecho]
CPU: 15 PID: 15257 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88003c53b760 ti: ffff88002b060000 task.ti: ffff88002b060000
RIP: 0010:[<ffffffffa16b1fb5>] [<ffffffffa16b1fb5>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff88002b063bc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802e592cd08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880087d648b8
RBP: ffff88002b063c40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff880025243078
R13: 0000000000000000 R14: ffff88009dfccf88 R15: ffff8802e5933a08
FS: 00007f5cd4f2d740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000010f3e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa169ce0c>] ll_dir_ioctl+0x6b9c/0x7750 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810a2009>] ? mod_timer+0x119/0x260
[<ffffffff810a2168>] ? add_timer+0x18/0x20
[<ffffffff810b10ab>] ? __queue_delayed_work+0x8b/0x1a0
[<ffffffffa03bea09>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff817e32dc>] ? _raw_spin_unlock_bh+0x1c/0x20
[<ffffffff811d92ef>] ? bdi_wakeup_thread_delayed+0x5f/0x70
[<ffffffff81277f01>] ? __mark_inode_dirty+0x161/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
LustreError: 27010:0:(osd_handler.c:226:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 27010:0:(osd_handler.c:226:osd_trans_start()) Skipped 6 previous similar messages
LustreError: 15086:0:(llite_lib.c:2021:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 15086:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff88027339ae98: inode [0x240002b15:0x918a:0x0] mdc close failed: rc = -28
Lustre: DEBUG MARKER: hardlinks
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1696fb5>] ll_migrate+0xa45/0xff0 [lustre]
PGD 2efed5067 PUD 1fecea067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy drm_panel_orientation_quirks virtio_blk serio_raw libata i2c_core [last unloaded: obdecho]
CPU: 2 PID: 12871 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802758c24f0 ti: ffff88029cf04000 task.ti: ffff88029cf04000
RIP: 0010:[<ffffffffa1696fb5>] [<ffffffffa1696fb5>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff88029cf07bc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801cb836988 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8803270f0a58
RBP: ffff88029cf07c40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff88001b2f7848
R13: 0000000000000000 R14: ffff88018d181738 R15: ffff8801cb830a88
FS: 00007ff0cb46e740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000010b378000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1681e0c>] ll_dir_ioctl+0x6b9c/0x7750 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810a2009>] ? mod_timer+0x119/0x260
[<ffffffff810a2168>] ? add_timer+0x18/0x20
[<ffffffff810b10ab>] ? __queue_delayed_work+0x8b/0x1a0
[<ffffffffa03d5a09>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff817e32dc>] ? _raw_spin_unlock_bh+0x1c/0x20
[<ffffffff811d92ef>] ? bdi_wakeup_thread_delayed+0x5f/0x70
[<ffffffff81277f01>] ? __mark_inode_dirty+0x161/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
LustreError: 10713:0:(osd_handler.c:226:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 12771:0:(llite_lib.c:2021:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 12771:0:(file.c:262:ll_close_inode_openhandle()) lustre-clilmv-ffff8802c573d3d8: inode [0x240002b15:0x8bfd:0x0] mdc close failed: rc = -28
Lustre: DEBUG MARKER: hardlinks
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate
LustreError: 10508:0:(osd_handler.c:226:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 10508:0:(osd_handler.c:226:osd_trans_start()) Skipped 19 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa168c0a5>] ll_migrate+0xa45/0xff0 [lustre]
PGD 28d087067 PUD 1d9d81067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks floppy ata_piix virtio_blk serio_raw libata i2c_core [last unloaded: obdecho]
CPU: 1 PID: 17794 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880040e10010 ti: ffff8802757c8000 task.ti: ffff8802757c8000
RIP: 0010:[<ffffffffa168c0a5>] [<ffffffffa168c0a5>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff8802757cbbc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88005165b088 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88026a7ea5c8
RBP: ffff8802757cbc40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff88029aacb208
R13: 0000000000000000 R14: ffff8802a5192128 R15: ffff88005165cd08
FS: 00007f422cb3c740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002d2256000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1676e0c>] ll_dir_ioctl+0x6b9c/0x7750 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810a2009>] ? mod_timer+0x119/0x260
[<ffffffff810a2168>] ? add_timer+0x18/0x20
[<ffffffff810b10ab>] ? __queue_delayed_work+0x8b/0x1a0
[<ffffffffa0472a09>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff817e32dc>] ? _raw_spin_unlock_bh+0x1c/0x20
[<ffffffff811d92ef>] ? bdi_wakeup_thread_delayed+0x5f/0x70
[<ffffffff81277f01>] ? __mark_inode_dirty+0x161/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
LustreError: 28452:0:(osd_handler.c:226:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
LustreError: 28452:0:(osd_handler.c:226:osd_trans_start()) Skipped 1 previous similar message
LustreError: 17610:0:(llite_lib.c:2028:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 17610:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff88008844a548: inode [0x240002b15:0x8ca5:0x0] mdc close failed: rc = -28
Lustre: DEBUG MARKER: hardlinks
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa168b0a5>] ll_migrate+0xa45/0xff0 [lustre]
PGD 2bf03e067 PUD 2ccd1c067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk serio_raw floppy i2c_core [last unloaded: obdecho]
CPU: 1 PID: 9068 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880123083760 ti: ffff88019bb48000 task.ti: ffff88019bb48000
RIP: 0010:[<ffffffffa168b0a5>] [<ffffffffa168b0a5>] ll_migrate+0xa45/0xff0 [lustre]
RSP: 0018:ffff88019bb4bbc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802f83bba08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8802bb5f7048
RBP: ffff88019bb4bc40 R08: 000000000000ffff R09: 000000000000ffff
R10: 0000000000000000 R11: 3d63722820676e69 R12: ffff8801d4109138
R13: 0000000000000000 R14: ffff8802b2b131b8 R15: ffff8802f83bd688
FS: 00007f3d159ce740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001af866000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1675e0c>] ll_dir_ioctl+0x6b9c/0x7750 [lustre]
[<ffffffff81402ab9>] ? snprintf+0x49/0x70
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa024ec07>] ? cfs_trace_unlock_tcd+0x27/0x70 [libcfs]
[<ffffffffa024f494>] ? libcfs_debug_msg+0x6d4/0xc20 [libcfs]
[<ffffffffa03afa19>] ? lprocfs_counter_add+0xe9/0x180 [obdclass]
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81277f48>] ? __mark_inode_dirty+0x1a8/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: DEBUG MARKER: touch/create
Lustre: DEBUG MARKER: hardlinks
LustreError: 17623:0:(osd_handler.c:226:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC: rc = -28
Lustre: DEBUG MARKER: cancel lru
Lustre: DEBUG MARKER: migrate

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa164a04c>] ll_migrate+0xa4c/0xf90 [lustre]
PGD 6b2c2067 PUD 53ac9067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy [last unloaded: obdecho]
CPU: 8 PID: 11727 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802a012a4f0 ti: ffff880072550000 task.ti: ffff880072550000
RIP: 0010:[<ffffffffa164a04c>] [<ffffffffa164a04c>] ll_migrate+0xa4c/0xf90 [lustre]
RSP: 0018:ffff880072553bc0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801d6b7c248 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88026f7aea78
RBP: ffff880072553c40 R08: 645f636f695f6c6c R09: 737265765f617461
R10: 617461645f636f69 R11: 6e6f69737265765f R12: ffff8800a6b7f078
R13: 0000000000000000 R14: ffff8802657a8108 R15: ffff8802ba8d0248
FS: 00007f168b73a740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000004408000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1635bcb>] ll_dir_ioctl+0x6a0b/0x7590 [lustre]
[<ffffffff81402544>] ? vsnprintf+0x234/0x6a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffffa0246487>] ? cfs_trace_unlock_tcd+0x27/0x70 [libcfs]
[<ffffffffa0246d30>] ? libcfs_debug_msg+0x6f0/0xa90 [libcfs]
[<ffffffff81416998>] ? debug_check_no_obj_freed+0x178/0x260
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81277f48>] ? __mark_inode_dirty+0x1a8/0x2c0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b89a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 4858:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0001: failed to start transaction due to ENOSPC
LustreError: 4858:0:(osd_handler.c:221:osd_trans_start()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160f13c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 1b1ce067 PUD 93b2f067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
LustreError: 26656:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8800668a53d8: inode [0x2000059f4:0x92f5:0x0] mdc close failed: rc = -28
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk i2c_core libata [last unloaded: obdecho]
CPU: 10 PID: 26641 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff880139e29280 ti: ffff8801a1ad0000 task.ti: ffff8801a1ad0000
RIP: 0010:[<ffffffffa160f13c>] [<ffffffffa160f13c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801a1ad3bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88009e525cc8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801040f3c78 RDI: ffff8801040f3c58
RBP: ffff8801a1ad3c48 R08: ffff88015c5225f8 R09: ffff8801a1ad3a60
R10: ffff88009e5266b8 R11: ffff88009e525978 R12: ffff880083ef47d8
R13: 0000000000000000 R14: ffff8802bc416158 R15: ffff8801618551f8
FS: 00007f8a6315f740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000099eee000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fc1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 12062:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 12062:0:(osd_handler.c:221:osd_trans_start()) Skipped 3 previous similar messages
LustreError: 26632:0:(llite_lib.c:1941:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 26632:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8800668a53d8: inode [0x2000059f4:0x92f5:0x0] mdc close failed: rc = -28
LustreError: 26641:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 26641:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fc13c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 26a09067 PUD f691067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy [last unloaded: obdecho]
CPU: 0 PID: 18046 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88024ac6b760 ti: ffff8800977fc000 task.ti: ffff8800977fc000
RIP: 0010:[<ffffffffa15fc13c>] [<ffffffffa15fc13c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8800977ffbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88026d5da7c8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802c3cdba98 RDI: ffff8802c3cdba78
RBP: ffff8800977ffc48 R08: ffff88006e85d158 R09: ffff8800977ffa60
R10: ffff88026d5d8358 R11: ffff88026d5dd978 R12: ffff8802798fea38
R13: 0000000000000000 R14: ffff880301946158 R15: ffff88000acb4758
FS: 00007facd0931740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000001cdea000 CR4: 00000000000007f0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15e91c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 18046:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 24508:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 18046:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 18095:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88029052ca88: inode [0x2000059f4:0x913b:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160c1bc>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 184318067 PUD 27bcfd067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy libata i2c_core [last unloaded: obdecho]
CPU: 15 PID: 19719 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802711d5c40 ti: ffff880108f64000 task.ti: ffff880108f64000
RIP: 0010:[<ffffffffa160c1bc>] [<ffffffffa160c1bc>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff880108f67bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88028063bef8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88032763ad78 RDI: ffff88032763ad58
RBP: ffff880108f67c48 R08: ffff8802840d4258 R09: ffff880108f67a60
R10: ffff8801822c5cc8 R11: ffff8801822c7a98 R12: ffff8801c0c50af8
R13: 0000000000000000 R14: ffff880308d40a58 R15: ffff8802c5de4758
FS: 00007fbff2e7c740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000153788000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15f91c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 19708:0:(llite_lib.c:1930:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 19708:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880153d5b7e8: inode [0x2000059f4:0x9195:0x0] mdc close failed: rc = -28
LustreError: 19708:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 19719:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 19719:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 19750:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880153d5b7e8: inode [0x2000059f4:0x9195:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f81bc>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 17c27b067 PUD 278ef1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk serio_raw i2c_core floppy [last unloaded: obdecho]
CPU: 7 PID: 24403 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88028d941280 ti: ffff88019d954000 task.ti: ffff88019d954000
RIP: 0010:[<ffffffffa15f81bc>] [<ffffffffa15f81bc>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88019d957bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88014133ea08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802fb8b1c08 RDI: ffff8802fb8b1be8
RBP: ffff88019d957c48 R08: ffff88008baaf138 R09: ffff88019d957a60
R10: ffff880141339a88 R11: ffff8801413393e8 R12: ffff8800aa318328
R13: 0000000000000000 R14: ffff8801b2fb0a58 R15: ffff88018dc5b218
FS: 00007f4dd5ce6740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002e2524000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15e51c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817d5802>] ? free_debug_processing+0x240/0x272
[<ffffffff8108982f>] ? __mmdrop+0x6f/0xa0
[<ffffffff8121e4c4>] ? __slab_free+0xa4/0x300
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff81416998>] ? debug_check_no_obj_freed+0x178/0x260
[<ffffffff8108982f>] ? __mmdrop+0x6f/0xa0
[<ffffffff8121e991>] ? kmem_cache_free+0x271/0x290
[<ffffffff8108982f>] ? __mmdrop+0x6f/0xa0
[<ffffffff810c844d>] ? finish_task_switch+0x15d/0x1b0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 24374:0:(llite_lib.c:1930:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 24374:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88013d262e98: inode [0x2000059f4:0x9197:0x0] mdc close failed: rc = -28
LustreError: 24403:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 24403:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 24441:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88013d262e98: inode [0x2000059f4:0x9197:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa161013c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 216080067 PUD 15a2f4067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy [last unloaded: obdecho]
CPU: 1 PID: 27642 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802c816dc40 ti: ffff88017c4f4000 task.ti: ffff88017c4f4000
RIP: 0010:[<ffffffffa161013c>] [<ffffffffa161013c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88017c4f7bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88009b33bba8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880272ffa038 RDI: ffff880272ffa018
RBP: ffff88017c4f7c48 R08: ffff880087c147f8 R09: ffff88017c4f7a60
R10: ffff88009b338008 R11: ffff88009b339dd8 R12: ffff8802592b1458
R13: 0000000000000000 R14: ffff8800a98ae158 R15: ffff880325022cc8
FS: 00007fca1599d740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000169dda000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fd1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810d7e3c>] ? set_next_entity+0x3c/0xe0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 27628:0:(llite_lib.c:1927:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 27628:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801e1cd4a88: inode [0x2000059f4:0x8ca4:0x0] mdc close failed: rc = -28
LustreError: 27628:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 27642:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 27642:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 27674:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801e1cd4a88: inode [0x2000059f4:0x8ca4:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa161e1bc>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 2e9e36067 PUD 14f3c2067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw virtio_blk i2c_core floppy [last unloaded: obdecho]
CPU: 9 PID: 8571 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88016b9e9280 ti: ffff8801f07e8000 task.ti: ffff8801f07e8000
RIP: 0010:[<ffffffffa161e1bc>] [<ffffffffa161e1bc>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801f07ebbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802a99a1738 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802ca745188 RDI: ffff8802ca745168
RBP: ffff8801f07ebc48 R08: ffff8800090beb98 R09: ffff8801f07eba60
R10: ffff8802a99a6018 R11: ffff8802a99a0008 R12: ffff8802568e9c28
R13: 0000000000000000 R14: ffff88023b119cd8 R15: ffff88026fee5748
FS: 00007f499cb7d740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002568be000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa160b1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 8540:0:(llite_lib.c:1930:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 8540:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801c9ef4a88: inode [0x2000059f4:0x924c:0x0] mdc close failed: rc = -28
LustreError: 8571:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 24494:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 24494:0:(osd_handler.c:221:osd_trans_start()) Skipped 11 previous similar messages
LustreError: 8571:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 8610:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801c9ef4a88: inode [0x2000059f4:0x924c:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa16001bc>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 2ddf30067 PUD 28d201067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk i2c_core serio_raw floppy libata [last unloaded: obdecho]
CPU: 14 PID: 32421 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88011fc39280 ti: ffff880181404000 task.ti: ffff880181404000
RIP: 0010:[<ffffffffa16001bc>] [<ffffffffa16001bc>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff880181407bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802f6b4c8e8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802d0e3e268 RDI: ffff8802d0e3e248
RBP: ffff880181407c48 R08: ffff880064c4fc78 R09: ffff880181407a60
R10: ffff8802f6b489f8 R11: ffff8802f6b4c598 R12: ffff8800828af078
R13: 0000000000000000 R14: ffff8802dbae7018 R15: ffff88012452c208
FS: 00007f8c8ec86740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000273e5e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15ed1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 32421:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 21510:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 32421:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 32495:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88031ccc9bf8: inode [0x2000059f4:0x9247:0x0] mdc close failed: rc = -28
LustreError: 32495:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ff17c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 26d6e067 PUD 81037067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata [last unloaded: obdecho]
CPU: 9 PID: 15881 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8801c4a71280 ti: ffff8800a760c000 task.ti: ffff8800a760c000
RIP: 0010:[<ffffffffa15ff17c>] [<ffffffffa15ff17c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8800a760fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801858a06a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880318d41718 RDI: ffff880318d416f8
RBP: ffff8800a760fc48 R08: ffff88031aa10f78 R09: ffff8800a760fa60
R10: ffff8801858a3858 R11: ffff8801858a3508 R12: ffff880273bb84b8
R13: 0000000000000000 R14: ffff88021cf6b898 R15: ffff88028b1fb768
FS: 00007f5bd1f5c740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000187c2000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15ec1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff8127c9c0>] ? generic_write_sync+0x60/0x60
[<ffffffff817e0a99>] ? schedule+0x29/0x70
[<ffffffff817dd739>] ? schedule_timeout+0x209/0x290
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff810b0d01>] ? __queue_work+0x151/0x420
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 15816:0:(llite_lib.c:1930:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 15816:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88005b699bf8: inode [0x200005224:0x9101:0x0] mdc close failed: rc = -28
LustreError: 15881:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 15881:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 15953:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88005b699bf8: inode [0x200005224:0x9101:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160d17c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 28acdb067 PUD 12bbcc067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: obdecho]
CPU: 6 PID: 28577 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8801fb29a4f0 ti: ffff8801f4538000 task.ti: ffff8801f4538000
RIP: 0010:[<ffffffffa160d17c>] [<ffffffffa160d17c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801f453bbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88013942b1b8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88020a39d0f8 RDI: ffff88020a39d0d8
RBP: ffff8801f453bc48 R08: ffff88031e3ced78 R09: ffff8801f453ba60
R10: ffff88013942ae68 R11: ffff88013942cf88 R12: ffff8802861d2d58
R13: 0000000000000000 R14: ffff8801ba8fa618 R15: ffff880017029cd8
FS: 00007f0ce0e42740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001d5ca0000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fa1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 11977:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 11977:0:(osd_handler.c:221:osd_trans_start()) Skipped 13 previous similar messages
LustreError: 28536:0:(llite_lib.c:1930:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 28536:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ef84138: inode [0x2000059f4:0x8b59:0x0] mdc close failed: rc = -28
LustreError: 28577:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 28577:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 28610:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88009ef84138: inode [0x2000059f4:0x8b59:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f358c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD a7fc4067 PUD 8b662067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core floppy libata [last unloaded: obdecho]
CPU: 2 PID: 29619 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88031d23a4f0 ti: ffff880048d74000 task.ti: ffff880048d74000
RIP: 0010:[<ffffffffa15f358c>] [<ffffffffa15f358c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff880048d77bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801a13a9dd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8800b46f2068 RDI: ffff8800b46f2048
RBP: ffff880048d77c48 R08: ffff880085745518 R09: ffff880048d77a60
R10: ffff88011b0fbef8 R11: ffff88011b0fd628 R12: ffff8802ea8a95e8
R13: 0000000000000000 R14: ffff880038ca1398 R15: ffff8801708e9788
FS: 00007f5565026740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000009b006000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15e01c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 29599:0:(llite_lib.c:1927:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 29599:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88027279efc8: inode [0x2000059f4:0x92f8:0x0] mdc close failed: rc = -28
LustreError: 29619:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 29619:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 20381:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 20381:0:(osd_handler.c:221:osd_trans_start()) Skipped 10 previous similar messages
LustreError: 29639:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88027279efc8: inode [0x2000059f4:0x92f8:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160d20c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 8b033067 PUD a2427067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core floppy libata [last unloaded: obdecho]
CPU: 15 PID: 2830 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88024bc45c40 ti: ffff8800854c8000 task.ti: ffff8800854c8000
RIP: 0010:[<ffffffffa160d20c>] [<ffffffffa160d20c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8800854cbbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88011706a128 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027c101d88 RDI: ffff88027c101d68
RBP: ffff8800854cbc48 R08: ffff88028b057318 R09: ffff8800854cba60
R10: ffff880117068358 R11: ffff88011706b1b8 R12: ffff8800828e5458
R13: 0000000000000000 R14: ffff880184e07018 R15: ffff8802cd258ce8
FS: 00007fc1c7dc6740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000000e6ce000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fa1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 2792:0:(llite_lib.c:1922:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 2792:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e63ba548: inode [0x2000059f4:0x93be:0x0] mdc close failed: rc = -28
LustreError: 2792:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 2830:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 2830:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 2880:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802e63ba548: inode [0x2000059f4:0x93be:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160f20c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 15b52a067 PUD 27904f067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: obdecho]
CPU: 9 PID: 5261 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88024c739280 ti: ffff88011eab4000 task.ti: ffff88011eab4000
RIP: 0010:[<ffffffffa160f20c>] [<ffffffffa160f20c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88011eab7bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801199ccc38 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027b23f168 RDI: ffff88027b23f148
RBP: ffff88011eab7c48 R08: ffff880287d75518 R09: ffff88011eab7a60
R10: ffff8801269b3ba8 R11: ffff8801269b1738 R12: ffff88015a6ee268
R13: 0000000000000000 R14: ffff8801f18a5d98 R15: ffff880275995748
FS: 00007f0784fbf740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002efed6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fc1c6>] ll_dir_ioctl+0x54e6/0x6430 [lustre]
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff8127c9c0>] ? generic_write_sync+0x60/0x60
[<ffffffff817e0a99>] ? schedule+0x29/0x70
[<ffffffff817dd739>] ? schedule_timeout+0x209/0x290
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff810b0d01>] ? __queue_work+0x151/0x420
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 5261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 5261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 5328:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cd2c8958: inode [0x200005224:0x9231:0x0] mdc close failed: rc = -28
LustreError: 5328:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa161041c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 2928ad067 PUD 186730067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw i2c_core virtio_blk libata floppy [last unloaded: obdecho]
CPU: 15 PID: 21803 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88028805c9d0 ti: ffff88025e4ac000 task.ti: ffff88025e4ac000
RIP: 0010:[<ffffffffa161041c>] [<ffffffffa161041c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88025e4afbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802f14a3508 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802be6f4808 RDI: ffff8802be6f47e8
RBP: ffff88025e4afc48 R08: ffff8800890ebe58 R09: ffff88025e4afa60
R10: ffff8802f14a4c38 R11: ffff8802f14a5628 R12: ffff880280999458
R13: 0000000000000000 R14: ffff8800786d1398 R15: ffff88028aa96c88
FS: 00007f385a83d740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000028df44000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fccf1>] ll_dir_ioctl+0x5e71/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 21803:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 8190:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 8190:0:(osd_handler.c:221:osd_trans_start()) Skipped 4 previous similar messages
LustreError: 21803:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 21837:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880283048008: inode [0x2000059f4:0x91fa:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa161135c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 2ce69e067 PUD 23cad9067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core floppy serio_raw virtio_blk libata [last unloaded: obdecho]
CPU: 2 PID: 10765 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802cfe1dc40 ti: ffff88020b0d0000 task.ti: ffff88020b0d0000
RIP: 0010:[<ffffffffa161135c>] [<ffffffffa161135c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88020b0d3bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801731ea128 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88030f93c048 RDI: ffff88030f93c028
RBP: ffff88020b0d3c48 R08: ffff88021372d8d8 R09: ffff88020b0d3a60
R10: ffff8801731ec248 R11: ffff8801731e9dd8 R12: ffff88026981b848
R13: 0000000000000000 R14: ffff88001b85cb18 R15: ffff88028e083768
FS: 00007fb833f78740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000274486000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15fdc7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 10735:0:(llite_lib.c:1921:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 10735:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f74cd3d8: inode [0x2000059f4:0x9403:0x0] mdc close failed: rc = -28
LustreError: 10765:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 10765:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 30277:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 30277:0:(osd_handler.c:221:osd_trans_start()) Skipped 7 previous similar messages
LustreError: 10784:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802f74cd3d8: inode [0x2000059f4:0x9403:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160d2ac>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 2e3fec067 PUD 1553f8067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
LustreError: 9580:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801bcf74a88: inode [0x2000059f4:0x9399:0x0] mdc close failed: rc = -28
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: obdecho]
CPU: 1 PID: 9543 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88031cdd1280 ti: ffff88010141c000 task.ti: ffff88010141c000
RIP: 0010:[<ffffffffa160d2ac>] [<ffffffffa160d2ac>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88010141fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801707e31b8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880139b36fe8 RDI: ffff880139b36fc8
RBP: ffff88010141fc48 R08: ffff8801c0ef5c98 R09: 0000000000000000
R10: ffff8800991da7c8 R11: ffff8800991db508 R12: ffff88000c19da98
R13: 0000000000000000 R14: ffff880061f25458 R15: ffff8800240c0248
FS: 00007f07f9a88740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000135b6c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15f9c7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 9536:0:(llite_lib.c:1894:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 9536:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8801bcf74a88: inode [0x2000059f4:0x9399:0x0] mdc close failed: rc = -28
LustreError: 9536:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 9543:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 9543:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa16082ac>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 1329d067 PUD 27453067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk floppy i2c_core serio_raw [last unloaded: obdecho]
CPU: 9 PID: 7924 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8800103649d0 ti: ffff880084120000 task.ti: ffff880084120000
RIP: 0010:[<ffffffffa16082ac>] [<ffffffffa16082ac>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff880084123bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88006e20cf88 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801020e5c28 RDI: ffff8801020e5c08
RBP: ffff880084123c48 R08: ffff880085f12d78 R09: ffff880084123a60
R10: ffff8800039b5628 R11: ffff8800039b06a8 R12: ffff88025f5f4e18
R13: 0000000000000000 R14: ffff8801ef471cd8 R15: ffff88005c190ce8
FS: 00007fa911fe1740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000000712d0000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15f4c7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 7924:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 31351:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 31351:0:(osd_handler.c:221:osd_trans_start()) Skipped 5 previous similar messages
LustreError: 7924:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 7978:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff880283845d28: inode [0x2000059f4:0x9282:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f72ac>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 161a22067 PUD 195034067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw libata i2c_core [last unloaded: obdecho]
CPU: 2 PID: 20619 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88012caedc40 ti: ffff8801349b0000 task.ti: ffff8801349b0000
RIP: 0010:[<ffffffffa15f72ac>] [<ffffffffa15f72ac>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801349b3bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88009a433ba8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880271782b08 RDI: ffff880271782ae8
RBP: ffff8801349b3c48 R08: ffff88012172b138 R09: ffff8801349b3a60
R10: ffff88013ca5a128 R11: ffff88013ca5ea08 R12: ffff880146bc4648
R13: 0000000000000000 R14: ffff8801a797a618 R15: ffff8802c8fbcca8
FS: 00007f3dcac71740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000015ccca000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15e3c7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 20619:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 16098:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 16098:0:(osd_handler.c:221:osd_trans_start()) Skipped 3 previous similar messages
LustreError: 20619:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 20702:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88027756efc8: inode [0x2000059f4:0x948b:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa160a2ac>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 293e05067 PUD 2d7144067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk i2c_core serio_raw libata floppy [last unloaded: obdecho]
CPU: 12 PID: 2313 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff880129f05c40 ti: ffff8801ebf5c000 task.ti: ffff8801ebf5c000
RIP: 0010:[<ffffffffa160a2ac>] [<ffffffffa160a2ac>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801ebf5fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88016ae927c8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88017b21d768 RDI: ffff88017b21d748
RBP: ffff8801ebf5fc48 R08: ffff88025dcab138 R09: ffff8801ebf5fa60
R10: ffff88026e8ec248 R11: ffff88026e8ed628 R12: ffff8801f2cf2ee8
R13: 0000000000000000 R14: ffff8802a54e4b18 R15: ffff88008c829238
FS: 00007f985c855740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000136d64000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15f6c7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff8127c9c0>] ? generic_write_sync+0x60/0x60
[<ffffffff817e0a99>] ? schedule+0x29/0x70
[<ffffffff817dd739>] ? schedule_timeout+0x209/0x290
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff810b0d01>] ? __queue_work+0x151/0x420
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 2313:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 2313:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 13838:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 13838:0:(osd_handler.c:221:osd_trans_start()) Skipped 6 previous similar messages
LustreError: 2423:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff88023acc4a88: inode [0x2000059f4:0x92e8:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f52ac>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 8a7b7067 PUD 87566067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix virtio_blk drm_panel_orientation_quirks serio_raw libata i2c_core floppy [last unloaded: obdecho]
CPU: 8 PID: 8337 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88026d618010 ti: ffff88002ac7c000 task.ti: ffff88002ac7c000
RIP: 0010:[<ffffffffa15f52ac>] [<ffffffffa15f52ac>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff88002ac7fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8800118248e8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031d15ebe8 RDI: ffff88031d15ebc8
RBP: ffff88002ac7fc48 R08: ffff8800759807f8 R09: ffff88002ac7fa60
R10: ffff880011820d48 R11: ffff880011825628 R12: ffff88002912c648
R13: 0000000000000000 R14: ffff880307518118 R15: ffff8803272871d8
FS: 00007f365b18b740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000002b3ae000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa15e1c7f>] ll_dir_ioctl+0x5ebf/0x74a0 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 14458:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 14458:0:(osd_handler.c:221:osd_trans_start()) Skipped 3 previous similar messages
LustreError: 8300:0:(llite_lib.c:1894:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 8300:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802bcb58958: inode [0x2000059f4:0x9367:0x0] mdc close failed: rc = -28
LustreError: 8337:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 8337:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 8417:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802bcb58958: inode [0x2000059f4:0x9367:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa165e24c>] ll_migrate+0xa1c/0xfa0 [lustre]
PGD 29d36e067 PUD 324b1a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: obdecho]
CPU: 8 PID: 16761 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802ca0d9280 ti: ffff8801312f0000 task.ti: ffff8801312f0000
RIP: 0010:[<ffffffffa165e24c>] [<ffffffffa165e24c>] ll_migrate+0xa1c/0xfa0 [lustre]
RSP: 0018:ffff8801312f3bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88010ce76a08 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880288865158 RDI: ffff880288865138
RBP: ffff8801312f3c48 R08: ffff8802c4d229b8 R09: ffff8801312f3a60
R10: ffff88010ce71a88 R11: ffff88010ce72128 R12: ffff8801c9a6df48
R13: 0000000000000000 R14: ffff88013b661398 R15: ffff8801a9caacc8
FS: 00007fe92d52a740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000019b800000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81242293>] ? __check_object_size+0x1c3/0x220
[<ffffffffa164b2a8>] ll_dir_ioctl+0x5618/0x65a0 [lustre]
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810d7e3c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810df63e>] ? pick_next_task_fair+0x65e/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 1585:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 1585:0:(osd_handler.c:221:osd_trans_start()) Skipped 11 previous similar messages
LustreError: 16733:0:(llite_lib.c:1894:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 16733:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802be6a4a88: inode [0x2000059f3:0x9336:0x0] mdc close failed: rc = -28
LustreError: 16733:0:(file.c:242:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 16761:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 16761:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 16822:0:(file.c:242:ll_close_inode_openhandle()) lustre-clilmv-ffff8802be6a4a88: inode [0x2000059f3:0x9336:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fbe7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 31f8c0067 PUD 27aaaa067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: obdecho]
CPU: 2 PID: 26756 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802c32624f0 ti: ffff88010f018000 task.ti: ffff88010f018000
RIP: 0010:[<ffffffffa15fbe7c>] [<ffffffffa15fbe7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88010f01bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880136013508 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802c58b95c8 RDI: ffff8802c58b95a8
RBP: ffff88010f01bc48 R08: ffff88002a1b25f8 R09: ffff88010f01ba68
R10: ffff8800075c6368 R11: ffff8800075c3ef8 R12: ffff880111d7f208
R13: 0000000000000000 R14: ffff880302e1c1d8 R15: ffff8802c9e8e1e8
FS: 00007fdf35fa9740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001508b2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e8b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff810b10ca>] ? __queue_delayed_work+0xaa/0x1a0
[<ffffffff810b1a51>] ? try_to_grab_pending+0xb1/0x180
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 28565:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 28565:0:(osd_handler.c:221:osd_trans_start()) Skipped 5 previous similar messages
LustreError: 26733:0:(llite_lib.c:1868:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 26733:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8801330d6fc8: inode [0x200005223:0x93e5:0x0] mdc close failed: rc = -28
LustreError: 26756:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 26756:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ffe7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 232b6a067 PUD 1461ed067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix i2c_core serio_raw libata virtio_blk floppy [last unloaded: obdecho]
CPU: 6 PID: 11690 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802c431b760 ti: ffff880167c50000 task.ti: ffff880167c50000
RIP: 0010:[<ffffffffa15ffe7c>] [<ffffffffa15ffe7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880167c53bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88000831b1b8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802ce249bd8 RDI: ffff8802ce249bb8
RBP: ffff880167c53c48 R08: ffff88010d1d8f78 R09: ffff880167c53a68
R10: ffff88018fb1e368 R11: ffff88018fb1ab18 R12: ffff8801cfa80198
R13: 0000000000000000 R14: ffff880304dee6d8 R15: ffff880283e551f8
FS: 00007fa6b4a0e740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000014aad2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15ecb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b630d>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c36c1>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b403e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff817df54c>] ? __mutex_unlock_slowpath+0xdc/0x190
[<ffffffff817df60e>] ? mutex_unlock+0xe/0x10
[<ffffffff81276d80>] ? sync_inodes_sb+0x160/0x430
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 11630:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 11630:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8802c0fa8008: inode [0x200005223:0x8be6:0x0] mdc close failed: rc = -28
LustreError: 11690:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 892:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 892:0:(osd_handler.c:221:osd_trans_start()) Skipped 7 previous similar messages
LustreError: 11690:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 11800:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8802c0fa8008: inode [0x200005223:0x8be6:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1602e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 9eb9b067 PUD 9cbd9067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks floppy ata_piix serio_raw virtio_blk i2c_core libata [last unloaded: obdecho]
CPU: 0 PID: 14684 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8801465aa4f0 ti: ffff8800104e0000 task.ti: ffff8800104e0000
RIP: 0010:[<ffffffffa1602e7c>] [<ffffffffa1602e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8800104e3bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88013347f3f8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802cb509568 RDI: ffff8802cb509548
RBP: ffff8800104e3c48 R08: ffff880238753c78 R09: ffff8800104e3a68
R10: ffff88013347a7c8 R11: ffff8801334793e8 R12: ffff8801417c5db8
R13: 0000000000000000 R14: ffff8801b0dbb898 R15: ffff8801b97ff728
FS: 00007f63f49a3740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000009bb88000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa15efb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff8127c9c0>] ? generic_write_sync+0x60/0x60
[<ffffffff817e0a99>] ? schedule+0x29/0x70
[<ffffffff817dd739>] ? schedule_timeout+0x209/0x290
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff810b0d01>] ? __queue_work+0x151/0x420
[<ffffffff810d5268>] ? __enqueue_entity+0x78/0x80
[<ffffffff810dbbab>] ? put_prev_entity+0x8b/0x400
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810d7e3c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 14637:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 14637:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880137d20008: inode [0x200005223:0x9323:0x0] mdc close failed: rc = -28
LustreError: 14684:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 14684:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 10993:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 10993:0:(osd_handler.c:221:osd_trans_start()) Skipped 11 previous similar messages
LustreError: 14702:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880137d20008: inode [0x200005223:0x9323:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1601e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 138bda067 PUD 16de8f067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: obdecho]
CPU: 14 PID: 30030 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88031ccd5c40 ti: ffff88017f9f4000 task.ti: ffff88017f9f4000
RIP: 0010:[<ffffffffa1601e7c>] [<ffffffffa1601e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88017f9f7bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88014e2b8008 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802941802c8 RDI: ffff8802941802a8
RBP: ffff88017f9f7c48 R08: ffff8802c7ee1158 R09: ffff88017f9f7a68
R10: ffff880197ad13e8 R11: ffff880197ad48e8 R12: ffff880170285f48
R13: 0000000000000000 R14: ffff880052192618 R15: ffff88008e9871d8
FS: 00007fe300637740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000156ffe000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15eeb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff817e05ca>] ? __schedule+0x32a/0x7d0
[<ffffffff8127c9c0>] ? generic_write_sync+0x60/0x60
[<ffffffff817e0a99>] ? schedule+0x29/0x70
[<ffffffff817dd739>] ? schedule_timeout+0x209/0x290
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff810b0d01>] ? __queue_work+0x151/0x420
[<ffffffff810d5268>] ? __enqueue_entity+0x78/0x80
[<ffffffff810dbbab>] ? put_prev_entity+0x8b/0x400
[<ffffffff810d5999>] ? pick_next_entity+0xa9/0x190
[<ffffffff810d7e3c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810df6e2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e329e>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c834d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff8127cb80>] ? do_fsync+0xa0/0xa0
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 29953:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 29953:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88001a7e9bf8: inode [0x2000059f3:0x9418:0x0] mdc close failed: rc = -28
LustreError: 29953:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 30030:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c sleeping for 5000ms
LustreError: 30030:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 32c awake
LustreError: 30076:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88001a7e9bf8: inode [0x2000059f3:0x9418:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fde7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 55691067 PUD ba234067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk libata i2c_core [last unloaded: obdecho]
CPU: 12 PID: 7076 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff880141895c40 ti: ffff880088944000 task.ti: ffff880088944000
RIP: 0010:[<ffffffffa15fde7c>] [<ffffffffa15fde7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880088947bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880266f486a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880070fd7748 RDI: ffff880070fd7728
RBP: ffff880088947c48 R08: ffff88024ad70438 R09: ffff880088947a68
R10: ffff880331403640 R11: ffff8801e929f748 R12: ffff88001e9dabc8
R13: 0000000000000000 R14: ffff8802ad58dd98 R15: ffff8803244aa228
FS: 00007f0b1565b740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000086182000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15eab82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 7049:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 7049:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880083e112a8: inode [0x2000059f3:0x92c0:0x0] mdc close failed: rc = -28
LustreError: 7049:0:(file.c:243:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1614e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 2f614c067 PUD 1f8b3a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk libata serio_raw floppy i2c_core [last unloaded: obdecho]
CPU: 15 PID: 30795 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802c9bd3760 ti: ffff8801eee20000 task.ti: ffff8801eee20000
RIP: 0010:[<ffffffffa1614e7c>] [<ffffffffa1614e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8801eee23bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802f8d5d628 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88004725a828 RDI: ffff88004725a808
RBP: ffff8801eee23c48 R08: ffff88028ac9ed78 R09: ffff8801eee23a68
R10: ffff8802f8d5f0a8 R11: ffff8802f8d5ed58 R12: ffff8801ac5d4c88
R13: 0000000000000000 R14: ffff8801f16ae6d8 R15: ffff8802b1d72228
FS: 00007ff303ec1740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000166fce000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1601b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 14396:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 14396:0:(osd_handler.c:221:osd_trans_start()) Skipped 12 previous similar messages
LustreError: 30780:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 30780:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88001b8392a8: inode [0x200005223:0x940e:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa161ce7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 161826067 PUD 274390067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk floppy libata i2c_core serio_raw [last unloaded: obdecho]
CPU: 5 PID: 27553 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff8802c6de1280 ti: ffff880153bd8000 task.ti: ffff880153bd8000
RIP: 0010:[<ffffffffa161ce7c>] [<ffffffffa161ce7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880153bdbbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880039610008 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802bd575618 RDI: ffff8802bd5755f8
RBP: ffff880153bdbc48 R08: ffff88003c8a1158 R09: ffff880153bdba68
R10: ffff880039613ba8 R11: ffff8800396152d8 R12: ffff8802b5cf7b68
R13: 0000000000000000 R14: ffff8802a4325d98 R15: ffff88008a635c98
FS: 00007f51bd85a740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001a5464000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1609b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 27496:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 27496:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880274de2e98: inode [0x2000059f3:0x9276:0x0] mdc close failed: rc = -28
LustreError: 27496:0:(file.c:243:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f0e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 446f2067 PUD 6d02e067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: obdecho]
CPU: 15 PID: 9030 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88026f73a4f0 ti: ffff880024820000 task.ti: ffff880024820000
RIP: 0010:[<ffffffffa15f0e7c>] [<ffffffffa15f0e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880024823bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802b622d978 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880247ce1838 RDI: ffff880247ce1818
RBP: ffff880024823c48 R08: ffff88027b97bc78 R09: ffff880024823a68
R10: ffff8802b622e368 R11: ffff8802b62289f8 R12: ffff8800163b23f8
R13: 0000000000000000 R14: ffff880070ffaf58 R15: ffff880283bee1e8
FS: 00007fc6978c8740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000003d820000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15ddb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 18207:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 8999:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 8999:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880054d537e8: inode [0x2000059f3:0x93c4:0x0] mdc close failed: rc = -28
LustreError: 8999:0:(file.c:243:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fce7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 26d909067 PUD 2912cc067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: obdecho]
CPU: 15 PID: 13482 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88024cf33760 ti: ffff88012a33c000 task.ti: ffff88012a33c000
RIP: 0010:[<ffffffffa15fce7c>] [<ffffffffa15fce7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88012a33fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880105aef3f8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880274fee7e8 RDI: ffff880274fee7c8
RBP: ffff88012a33fc48 R08: ffff8802869da418 R09: ffff88012a33fa68
R10: ffff880105aec248 R11: ffff880105aefa98 R12: ffff880073256a38
R13: 0000000000000000 R14: ffff8801fda2c1d8 R15: ffff8802c61aec88
FS: 00007f906f903740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000028145c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e9b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 13461:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 13461:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8801b1222e98: inode [0x2000059f3:0x94b3:0x0] mdc close failed: rc = -28
LustreError: 13461:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1606e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 293f66067 PUD 285c80067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk libata i2c_core serio_raw [last unloaded: obdecho]
CPU: 14 PID: 30334 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.15.0-1.module_el8.6.0+1087+b42c8331 04/01/2014
task: ffff88001e02b760 ti: ffff88016aa54000 task.ti: ffff88016aa54000
RIP: 0010:[<ffffffffa1606e7c>] [<ffffffffa1606e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88016aa57bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88012d459dd8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802cdcc0d68 RDI: ffff8802cdcc0d48
RBP: ffff88016aa57c48 R08: ffff8802fac927d8 R09: ffff88016aa57a68
R10: ffff88012d458d48 R11: ffff88012d45cc38 R12: ffff8802f7221db8
R13: 0000000000000000 R14: ffff8802f7531cd8 R15: ffff880013d371d8
FS: 00007f89def31740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000014fd46000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15f3b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 30289:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 30289:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880071b61bf8: inode [0x2000059f3:0x9456:0x0] mdc close failed: rc = -28
LustreError: 30289:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1601e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 248434067 PUD 2ec1fb067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy [last unloaded: obdecho]
CPU: 10 PID: 24137 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880283c0b760 ti: ffff8801ea0e4000 task.ti: ffff8801ea0e4000
RIP: 0010:[<ffffffffa1601e7c>] [<ffffffffa1601e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8801ea0e7bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801ff762128 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880323b391f8 RDI: ffff880323b391d8
RBP: ffff8801ea0e7c48 R08: ffff88017c0eeb98 R09: ffff8801ea0e7a68
R10: ffff8801ff7631b8 R11: ffff8801ff766d58 R12: ffff8801d78d8e18
R13: 0000000000000000 R14: ffff8802d3934b18 R15: ffff8801856bb218
FS: 00007fdb7e437740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001761f4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15eeb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 24107:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 24107:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88000fa237e8: inode [0x200005223:0x94a6:0x0] mdc close failed: rc = -28
LustreError: 24107:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1609e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 15d162067 PUD 28e662067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata floppy i2c_core [last unloaded: obdecho]
CPU: 7 PID: 331 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8802709c8010 ti: ffff88018548c000 task.ti: ffff88018548c000
RIP: 0010:[<ffffffffa1609e7c>] [<ffffffffa1609e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88018548fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88005480a7c8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880289a6ebb8 RDI: ffff880289a6eb98
RBP: ffff88018548fc48 R08: ffff8802cd24b8b8 R09: ffff88018548fa68
R10: ffff88005480e368 R11: ffff8800548089f8 R12: ffff8802e5327cf8
R13: 0000000000000000 R14: ffff8800b2331398 R15: ffff880290a68ce8
FS: 00007f9dfd235740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000273210000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15f6b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 305:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 305:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8800192bc138: inode [0x2000059f3:0x96da:0x0] mdc close failed: rc = -28
LustreError: 305:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f7e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 1e117b067 PUD 296f91067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw floppy i2c_core libata [last unloaded: obdecho]
CPU: 7 PID: 5157 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88014b789280 ti: ffff8801a619c000 task.ti: ffff8801a619c000
RIP: 0010:[<ffffffffa15f7e7c>] [<ffffffffa15f7e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8801a619fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801ab7206a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880288958178 RDI: ffff880288958158
RBP: ffff8801a619fc48 R08: ffff880277dc4bb8 R09: ffff8801a619fa68
R10: ffff880331403640 R11: ffff880014f0cc38 R12: ffff88000ec41a98
R13: 0000000000000000 R14: ffff8801e9938a58 R15: ffff8802c2e4bcb8
FS: 00007f6fc314d740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000010d0ca000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e4b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 5131:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 5131:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8802c7f9c138: inode [0x200005223:0x92a2:0x0] mdc close failed: rc = -28
LustreError: 5131:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1625e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 2c4021067 PUD 31f1c8067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata [last unloaded: obdecho]
CPU: 5 PID: 2240 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880041460010 ti: ffff880125bf8000 task.ti: ffff880125bf8000
RIP: 0010:[<ffffffffa1625e7c>] [<ffffffffa1625e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880125bfbbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802da972128 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802c64e70a8 RDI: ffff8802c64e7088
RBP: ffff880125bfbc48 R08: ffff8801c8ba3138 R09: ffff880125bfba68
R10: ffff8801abc989f8 R11: ffff8801abc99dd8 R12: ffff8802076ae718
R13: 0000000000000000 R14: ffff8802527166d8 R15: ffff8800a82db768
FS: 00007fa30d893740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000020b274000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1612b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 2212:0:(llite_lib.c:1873:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 2212:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88023e1bdd28: inode [0x200005223:0x94c5:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1615e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 1f8104067 PUD 2930a4067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: obdecho]
CPU: 8 PID: 6392 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8802d7cc3760 ti: ffff880278208000 task.ti: ffff880278208000
RIP: 0010:[<ffffffffa1615e7c>] [<ffffffffa1615e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88027820bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801925466b8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027c7e2188 RDI: ffff88027c7e2168
RBP: ffff88027820bc48 R08: ffff8802214996f8 R09: ffff88027820ba68
R10: ffff880192546d58 R11: ffff880192545cc8 R12: ffff8801a23e5908
R13: 0000000000000000 R14: ffff88023783c1d8 R15: ffff880293d31cd8
FS: 00007fdac7e59740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000191650000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1602b82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 6262:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 6262:0:(osd_handler.c:221:osd_trans_start()) Skipped 4 previous similar messages
LustreError: 6341:0:(llite_lib.c:1873:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 6341:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff880160b512a8: inode [0x2000059f3:0x8d39:0x0] mdc close failed: rc = -28
LustreError: 6341:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1602e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 11a06d067 PUD 137aa3067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata i2c_core floppy [last unloaded: obdecho]
CPU: 8 PID: 22487 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88028d2149d0 ti: ffff8801a1d08000 task.ti: ffff8801a1d08000
RIP: 0010:[<ffffffffa1602e7c>] [<ffffffffa1602e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8801a1d0bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801d6c8b508 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880217601198 RDI: ffff880217601178
RBP: ffff8801a1d0bc48 R08: ffff88017e422f58 R09: ffff8801a1d0ba68
R10: ffff8801d6c8c248 R11: ffff8801d6c8b1b8 R12: ffff88016367ad58
R13: 0000000000000000 R14: ffff88020bcb2618 R15: ffff88026fff61e8
FS: 00007f960d284740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002c06ce000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15efb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 24802:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 24802:0:(osd_handler.c:221:osd_trans_start()) Skipped 1 previous similar message
LustreError: 22468:0:(llite_lib.c:1873:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 22468:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88001fb0e678: inode [0x200005223:0x9376:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1652e7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 24563e067 PUD 23cdf7067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core floppy libata [last unloaded: obdecho]
CPU: 7 PID: 31999 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8802c6d3c9d0 ti: ffff880273c28000 task.ti: ffff880273c28000
RIP: 0010:[<ffffffffa1652e7c>] [<ffffffffa1652e7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880273c2bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88016e4e3ba8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88009e404bd8 RDI: ffff88009e404bb8
RBP: ffff880273c2bc48 R08: ffff8800ace36d78 R09: ffff880273c2ba68
R10: ffff88016e4e3858 R11: ffff88016e4e6018 R12: ffff8801392e7208
R13: 0000000000000000 R14: ffff88026d9ddd98 R15: ffff880143da0ce8
FS: 00007f1571f52740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000013970c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa163fb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 31964:0:(llite_lib.c:1873:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 31964:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff88006df55d28: inode [0x200005223:0x9466:0x0] mdc close failed: rc = -28
LustreError: 31964:0:(file.c:243:ll_close_inode_openhandle()) Skipped 2 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ffe7c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 10745e067 PUD 13e57a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks virtio_blk ata_piix serio_raw i2c_core libata floppy [last unloaded: obdecho]
CPU: 1 PID: 5293 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880089a8c9d0 ti: ffff88019047c000 task.ti: ffff88019047c000
RIP: 0010:[<ffffffffa15ffe7c>] [<ffffffffa15ffe7c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88019047fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88006493ab18 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027d6e9048 RDI: ffff88027d6e9028
RBP: ffff88019047fc48 R08: ffff88009bf309d8 R09: ffff88019047fa68
R10: ffff88006493bba8 R11: ffff88006493ea08 R12: ffff88027bf13398
R13: 0000000000000000 R14: ffff8801631741d8 R15: ffff880049f39238
FS: 00007fb903ff8740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002d8b88000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15ecb82>] ll_dir_ioctl+0x5d12/0x7290 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 5251:0:(llite_lib.c:1873:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 5251:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8800ae51b7e8: inode [0x2000059f3:0x92e8:0x0] mdc close failed: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ec01c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 2e8e29067 PUD 2c5a99067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw libata i2c_core [last unloaded: obdecho]
CPU: 11 PID: 5109 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880016fb9280 ti: ffff8802a0248000 task.ti: ffff8802a0248000
RIP: 0010:[<ffffffffa15ec01c>] [<ffffffffa15ec01c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8802a024bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8800230470a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027843cc98 RDI: ffff88027843cc78
RBP: ffff8802a024bc48 R08: ffff880255884438 R09: ffff8802a024ba68
R10: ffff880023043858 R11: ffff880023046018 R12: ffff88014352a3f8
R13: 0000000000000000 R14: ffff880021b92f58 R15: ffff88008cb27728
FS: 00007fe07e0b3740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000027dd78000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15d8ece>] ll_dir_ioctl+0x618e/0x7560 [lustre]
[<ffffffff812341c0>] ? __mem_cgroup_commit_charge+0xc0/0x390
[<ffffffff817e324e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff811ea55d>] ? handle_pte_fault+0x64d/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 5071:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28
LustreError: 5071:0:(file.c:243:ll_close_inode_openhandle()) lustre-clilmv-ffff8802c3efca88: inode [0x2000059f3:0x92ef:0x0] mdc close failed: rc = -28
LustreError: 5071:0:(file.c:243:ll_close_inode_openhandle()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15e718c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 98170067 PUD 12886067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw floppy i2c_core virtio_blk libata [last unloaded: obdecho]
CPU: 10 PID: 25047 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880138b2c9d0 ti: ffff8802eee00000 task.ti: ffff8802eee00000
RIP: 0010:[<ffffffffa15e718c>] [<ffffffffa15e718c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8802eee03bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88013252c248 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802765021e8 RDI: ffff8802765021c8
RBP: ffff8802eee03c48 R08: ffff8802859b5338 R09: ffff8802eee03a68
R10: ffff880174aaea08 R11: ffff880174aae368 R12: ffff8802c01c6a38
R13: 0000000000000000 R14: ffff8801d70f9cd8 R15: ffff880130cd0ce8
FS: 00007f5b8f6c7740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000003d2f2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15d40a6>] ll_dir_ioctl+0x5c46/0x7080 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 25028:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f99bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 31ae2f067 PUD 12de4a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: obdecho]
CPU: 0 PID: 28816 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880290328010 ti: ffff88019f530000 task.ti: ffff88019f530000
RIP: 0010:[<ffffffffa15f99bc>] [<ffffffffa15f99bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88019f533bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880134e4cc38 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802914e7508 RDI: ffff8802914e74e8
RBP: ffff88019f533c48 R08: ffff8801e8448f78 R09: ffff88019f533a68
R10: ffff880134e486a8 R11: ffff880134e4f0a8 R12: ffff880098fc8008
R13: 0000000000000000 R14: ffff88003cf20a58 R15: ffff8802c0e78ce8
FS: 00007f4906fb8740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000013f0c0000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa15e6bf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 28766:0:(llite_lib.c:1871:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ff9bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 14b6b0067 PUD 2874e1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks serio_raw virtio_blk i2c_core ata_piix floppy libata [last unloaded: obdecho]
CPU: 15 PID: 19452 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88032901dc40 ti: ffff8802f4e44000 task.ti: ffff8802f4e44000
RIP: 0010:[<ffffffffa15ff9bc>] [<ffffffffa15ff9bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8802f4e47bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88011dc27a98 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8800ae7c42b8 RDI: ffff8800ae7c4298
RBP: ffff8802f4e47c48 R08: ffff88027791ef58 R09: ffff8802f4e47a68
R10: ffff88014b0bb1b8 R11: ffff88014b0bb858 R12: ffff880164850968
R13: 0000000000000000 R14: ffff8801b5c6dd98 R15: ffff88008b497c78
FS: 00007f86fca3a740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000159100000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15ecbf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 7577:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 7577:0:(osd_handler.c:221:osd_trans_start()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15f39bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 27982d067 PUD 2f1bf2067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk i2c_core libata [last unloaded: obdecho]
CPU: 7 PID: 32521 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8802ee74b760 ti: ffff88027b65c000 task.ti: ffff88027b65c000
RIP: 0010:[<ffffffffa15f39bc>] [<ffffffffa15f39bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88027b65fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880084382e68 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88008a0c0c18 RDI: ffff88008a0c0bf8
RBP: ffff88027b65fc48 R08: ffff8802594fb8b8 R09: ffff88027b65fa68
R10: ffff8800843870a8 R11: ffff880084383858 R12: ffff8802beb344b8
R13: 0000000000000000 R14: ffff880069c5f018 R15: ffff880324188798
FS: 00007fafe3de8740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002bb950000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e0bf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 5488:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 32493:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ff9bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 92ae067 PUD 6e93f067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: obdecho]
CPU: 10 PID: 30231 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88027e4b24f0 ti: ffff88006804c000 task.ti: ffff88006804c000
RIP: 0010:[<ffffffffa15ff9bc>] [<ffffffffa15ff9bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88006804fbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88023b13c598 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88009e7431d8 RDI: ffff88009e7431b8
RBP: ffff88006804fc48 R08: ffff8801ee92e7d8 R09: ffff88006804fa68
R10: ffff880331403640 R11: ffff880288419098 R12: ffff8801e9c21f48
R13: 0000000000000000 R14: ffff8803212d1cd8 R15: ffff8801498ec208
FS: 00007f411ae49740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000004ab18000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15ecbf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 4658:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 30206:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15e69bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 171c27067 PUD 2e3607067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm serio_raw ata_piix drm_panel_orientation_quirks i2c_core virtio_blk libata floppy [last unloaded: obdecho]
CPU: 1 PID: 29062 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88005086a4f0 ti: ffff880176ee8000 task.ti: ffff880176ee8000
RIP: 0010:[<ffffffffa15e69bc>] [<ffffffffa15e69bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880176eebbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88013e843ef8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802aef53ac8 RDI: ffff8802aef53aa8
RBP: ffff880176eebc48 R08: ffff880062546d78 R09: ffff880176eeba68
R10: ffff880331403640 R11: ffff8801227cd2d8 R12: ffff8801346fd5e8
R13: 0000000000000000 R14: ffff88016d90f018 R15: ffff88027a068248
FS: 00007fe704dbf740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000106b72000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15d3bf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 29030:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15d99bc>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 167f26067 PUD 2bc3fc067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw i2c_core virtio_blk libata [last unloaded: obdecho]
CPU: 0 PID: 8955 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8801ab09a4f0 ti: ffff880139110000 task.ti: ffff880139110000
RIP: 0010:[<ffffffffa15d99bc>] [<ffffffffa15d99bc>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880139113bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880162414248 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880284f2fa58 RDI: ffff880284f2fa38
RBP: ffff880139113c48 R08: ffff880244aeab98 R09: ffff880139113a68
R10: ffff8802b9ee5628 R11: ffff8802b9ee7a98 R12: ffff88013fcf5458
R13: 0000000000000000 R14: ffff8802de805458 R15: ffff880074520ce8
FS: 00007fcf368ff740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000028a29c000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa15c6bf7>] ll_dir_ioctl+0x5dc7/0x6ee0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 17954:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 8924:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fbc6c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 2c5da2067 PUD 170bf8067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk floppy serio_raw i2c_core libata [last unloaded: obdecho]
CPU: 10 PID: 25579 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8803296e9280 ti: ffff880241e10000 task.ti: ffff880241e10000
RIP: 0010:[<ffffffffa15fbc6c>] [<ffffffffa15fbc6c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff880241e13bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8801ff73c248 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880282b32c88 RDI: ffff880282b32c68
RBP: ffff880241e13c48 R08: ffff880289b35e78 R09: ffff880241e13a68
R10: ffff8801ff73ae68 R11: ffff8801ff73b508 R12: ffff880175816268
R13: 0000000000000000 R14: ffff88001965cb18 R15: ffff880207706738
FS: 00007f342dcca740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000298c0e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e8b86>] ll_dir_ioctl+0x5c46/0x7080 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 18534:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 18534:0:(osd_handler.c:221:osd_trans_start()) Skipped 1 previous similar message
LustreError: 25563:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15fac6c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 2cea23067 PUD 1a7a92067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata virtio_blk floppy i2c_core [last unloaded: obdecho]
CPU: 4 PID: 25266 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88032000c9d0 ti: ffff88016dd38000 task.ti: ffff88016dd38000
RIP: 0010:[<ffffffffa15fac6c>] [<ffffffffa15fac6c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88016dd3bbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88001f38cf88 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88027d13b0b8 RDI: ffff88027d13b098
RBP: ffff88016dd3bc48 R08: ffff88000b9f29b8 R09: ffff88016dd3ba68
R10: ffff88001f38ed58 R11: ffff88001f388d48 R12: ffff88016012ea38
R13: 0000000000000000 R14: ffff8801b5bf0a58 R15: ffff880276fe4208
FS: 00007f479acf3740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000002c99ba000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15e7b86>] ll_dir_ioctl+0x5c46/0x7080 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 30819:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 30819:0:(osd_handler.c:221:osd_trans_start()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15eec6c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 18e33b067 PUD 2f84b1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks libata i2c_core virtio_blk serio_raw floppy [last unloaded: obdecho]
CPU: 14 PID: 18832 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff88026e64dc40 ti: ffff88011be90000 task.ti: ffff88011be90000
RIP: 0010:[<ffffffffa15eec6c>] [<ffffffffa15eec6c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88011be93bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880130b7bba8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8802c6fced08 RDI: ffff8802c6fcece8
RBP: ffff88011be93c48 R08: ffff88026dee9338 R09: ffff88011be93a68
R10: ffff880130b789f8 R11: ffff880130b7c598 R12: ffff88019c74e718
R13: 0000000000000000 R14: ffff8801027b66d8 R15: ffff8800832d4758
FS: 00007fc7a5c16740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000019c750000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15dbb86>] ll_dir_ioctl+0x5c46/0x7080 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 19403:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 19403:0:(osd_handler.c:221:osd_trans_start()) Skipped 4 previous similar messages
LustreError: 18801:0:(llite_lib.c:1848:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15eef9c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 293e82067 PUD 13c40a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw libata i2c_core floppy [last unloaded: obdecho]
CPU: 9 PID: 13185 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff8801fb4c8010 ti: ffff8802878f4000 task.ti: ffff8802878f4000
RIP: 0010:[<ffffffffa15eef9c>] [<ffffffffa15eef9c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff8802878f7bd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88016a1d2e68 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88009fc6dc28 RDI: ffff88009fc6dc08
RBP: ffff8802878f7c48 R08: ffff880295ebef58 R09: ffff8802878f7a68
R10: ffff880331403640 R11: ffff88018dd0cc38 R12: ffff8802e07e0c88
R13: 0000000000000000 R14: ffff8800458fdd98 R15: ffff8802daeeec88
FS: 00007f7d32473740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 00000001a0678000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15dca2d>] ll_dir_ioctl+0x542d/0x61e0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 21273:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 21273:0:(osd_handler.c:221:osd_trans_start()) Skipped 2 previous similar messages
LustreError: 13157:0:(llite_lib.c:1843:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa15ebf9c>] ll_migrate+0x9ec/0xf50 [lustre]
PGD 11d069067 PUD 122724067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw i2c_core libata virtio_blk [last unloaded: obdecho]
CPU: 14 PID: 11829 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.5.0+746+bbd5d70c 04/01/2014
task: ffff880183e88010 ti: ffff88012aaa8000 task.ti: ffff88012aaa8000
RIP: 0010:[<ffffffffa15ebf9c>] [<ffffffffa15ebf9c>] ll_migrate+0x9ec/0xf50 [lustre]
RSP: 0018:ffff88012aaabbd0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8800859f4c38 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880068d4e7b8 RDI: ffff880068d4e798
RBP: ffff88012aaabc48 R08: ffff88016d2256f8 R09: ffff88012aaaba68
R10: ffff8800859f3ef8 R11: ffff8800859f27c8 R12: ffff8800202136b8
R13: 0000000000000000 R14: ffff8801640a9398 R15: ffff880199b42cc8
FS: 00007fea6d36e740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 000000024a406000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa15d9a2d>] ll_dir_ioctl+0x542d/0x61e0 [lustre]
[<ffffffff811b518b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e5e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1ff>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff811ecf9d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b47d>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8125b7d1>] SyS_ioctl+0xa1/0xc0
[<ffffffff817edf49>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LustreError: 17085:0:(osd_handler.c:221:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC
LustreError: 11797:0:(llite_lib.c:1843:ll_md_setattr()) md_setattr fails: rc = -28

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa104de52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 800000008c2c5067 PUD 81ad1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel drm_panel_orientation_quirks virtio_blk libata serio_raw i2c_core floppy
CPU: 11 PID: 30710 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880326a7c9d0 ti: ffff880265bbc000 task.ti: ffff880265bbc000
RIP: 0010:[<ffffffffa104de52>] [<ffffffffa104de52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff880265bbfbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880242aa4968 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880321fab758 RDI: ffff880321fab738
RBP: ffff880265bbfc48 R08: ffff88025f601158 R09: 0000000000000001
R10: 0000000000000000 R11: ffff880265bbf5e6 R12: 0000000000000000
R13: ffff880261382128 R14: ffff8800911a9cd8 R15: 0000000000000030
FS: 00007ff7995e6740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000267ec2000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa103b561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8115260f>] ? delayacct_end+0x8f/0xb0
[<ffffffff811b62dd>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c3691>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b400e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81264d2b>] ? iput+0x3b/0x180
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: lustre-OST0000-osc-ffff880298df53d8: disconnect after 21s idle
Lustre: 11097:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 510 < left 6612, rollback = 9
Lustre: 11097:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 388/1552/0, destroy: 1/4/0
Lustre: 11097:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 392/148/0
Lustre: 11097:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1941/16694/0, punch: 0/0/0, quota 6/6/0
Lustre: 11097:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 389/6612/0, delete: 2/5/1
Lustre: 11097:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0
Lustre: 11097:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 509 < left 16694, rollback = 7
Lustre: 11113:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 511 < left 16694, rollback = 7
Lustre: 11113:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 388/1552/0, destroy: 1/4/0
Lustre: 11113:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11113:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 388/0/0
Lustre: 11113:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11113:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1941/16694/0, punch: 0/0/0, quota 6/6/0
Lustre: 11113:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11113:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 388/6596/0, delete: 2/5/0
Lustre: 11113:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11113:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/1
Lustre: 11113:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11113:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 16693, rollback = 7
Lustre: 11113:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 388/1552/0, destroy: 1/4/0
Lustre: 16626:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 582, rollback = 2
Lustre: 16626:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 6/582/0
Lustre: 16626:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 16626:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 16626:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 16626:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 16626:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 16626:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 16626:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 11113:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 474 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 2 PID: 11113 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel drm_panel_orientation_quirks virtio_blk libata serio_raw i2c_core floppy
CPU: 2 PID: 11113 Comm: mdt_rdpg01_001 Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0be54ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0bd69eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0bd9417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0bf9aa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0bf9ad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0c8ed08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04d859d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04dccf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0e46af4>] mdt_mfd_close+0x1f4/0x870 [mdt]
[<ffffffffa0e4ce14>] mdt_close_internal+0xb4/0x240 [mdt]
[<ffffffffa0e4d224>] mdt_close+0x284/0x850 [mdt]
[<ffffffffa07df2d4>] ? tgt_request_preprocess.isra.27+0xf4/0x850 [ptlrpc]
[<ffffffffa07e026e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032d45e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa078b990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa078d529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffffa078c950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 28bda789721aa7ff ]---
Lustre: 4129:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88028a469940 x1710331742644800/t0(0) o10->lustre-OST0001-osc-ffff880298df53d8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631099681 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 521, rollback = 2
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 11089:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/3, destroy: 0/0/0
Lustre: 11089:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 2321 previous similar messages
Lustre: 11089:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/521/0
Lustre: 11089:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 2871 previous similar messages
Lustre: 11089:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 11089:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 2871 previous similar messages
Lustre: 11089:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 11089:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 2869 previous similar messages
Lustre: 11089:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 11089:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 2869 previous similar messages
LustreError: 12289:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 12289:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
Lustre: 16601:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 1903, rollback = 7
Lustre: 16601:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2867 previous similar messages
Lustre: 18306:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880287b9d180 x1710331742999936/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:712/0 lens 440/0 e 0 to 0 dl 1631099692 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 11104:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 1746, rollback = 7
Lustre: 11104:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1893 previous similar messages
Lustre: 11104:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11104:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1897 previous similar messages
Lustre: 11104:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11104:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1897 previous similar messages
Lustre: 11104:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1737/1746/0, punch: 0/0/0, quota 3/3/0
Lustre: 11104:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1897 previous similar messages
Lustre: 11104:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11104:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1897 previous similar messages
Lustre: 11104:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11104:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1897 previous similar messages
LustreError: 18301:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 6/5s ago req@ffff880275fc1f80 x1710331742994176/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:712/0 lens 440/0 e 0 to 0 dl 1631099692 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18330:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6/5s); client may timeout req@ffff88027548ea80 x1710331742994432/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:712/0 lens 440/0 e 0 to 0 dl 1631099692 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 18301:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 2 previous similar messages
LustreError: 18301:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 6/5s ago req@ffff88028018de00 x1710331742999616/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:712/0 lens 440/0 e 0 to 0 dl 1631099692 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 18301:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 14 previous similar messages
Lustre: 18301:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6/6s); client may timeout req@ffff88028018de00 x1710331742999616/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:712/0 lens 440/0 e 0 to 0 dl 1631099692 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18301:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 16 previous similar messages
Lustre: 18421:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880279530680 x1710331743026752/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:717/0 lens 440/0 e 0 to 0 dl 1631099697 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18421:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages
LustreError: 18350:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 6/7s ago req@ffff88027ec78040 x1710331743023552/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:717/0 lens 440/0 e 0 to 0 dl 1631099697 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18315:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6/7s); client may timeout req@ffff88027ec7bec0 x1710331743023744/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:717/0 lens 440/0 e 0 to 0 dl 1631099697 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 18350:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 17 previous similar messages
Lustre: 4139:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631099686/real 1631099686] req@ffff880277e38680 x1710331742994176/t0(0) o1->lustre-OST0003-osc-ffff880298dfae98@0@lo:28/4 lens 440/432 e 0 to 1 dl 1631099695 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0003: Client bc1e05b8-368a-4968-928a-1b45107d748e (at 0@lo) reconnecting
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: 4140:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631099691/real 1631099691] req@ffff880085e31300 x1710331743023552/t0(0) o1->lustre-OST0002-osc-ffff880298dfae98@0@lo:28/4 lens 440/432 e 0 to 1 dl 1631099703 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: lustre-OST0002-osc-ffff880298dfae98: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0000: Client bc1e05b8-368a-4968-928a-1b45107d748e (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffff880298dfae98: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: 17011:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631099668/real 1631099668] req@ffff8802849b9f80 x1710331742473088/t0(0) o101->lustre-MDT0000-mdc-ffff880298df53d8@0@lo:12/10 lens 376/32368 e 0 to 1 dl 1631099712 ref 2 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'dd.0'
Lustre: 17011:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff880298df53d8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000: Client 1154ae0e-996a-42a4-bcbf-d44ba44315ac (at 0@lo) reconnecting
Lustre: Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 18407:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 699, rollback = 2
Lustre: 18407:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 18407:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0
Lustre: 18407:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1736 previous similar messages
Lustre: 18407:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 9/699/0
Lustre: 18407:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1736 previous similar messages
Lustre: 18407:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 18407:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1736 previous similar messages
Lustre: 18407:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/2, delete: 1/1/0
Lustre: 18407:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1736 previous similar messages
Lustre: 18407:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 18407:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1736 previous similar messages
sched: RT throttling activated
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 509 < left 22405, rollback = 9
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1317/5268/0, destroy: 1/4/0
Lustre: 11089:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 1321/148/0
Lustre: 11089:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 6586/56641/0, punch: 0/0/0, quota 6/6/0
Lustre: 11089:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1318/22405/0, delete: 2/5/1
Lustre: 11089:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 11089:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 507 < left 56641, rollback = 7
Lustre: 11089:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1736 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff88008ab587c0/0x4105371653f934af lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x5:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x4105371653f934a8 expref: 2784 pid: 17754 timeout: 451 lvb_type: 0
LustreError: 18382:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880275eb7700 x1710331746390528/t0(0) o106->lustre-OST0001@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 18348:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802652e3ec0 x1710331746405440/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0001-osc-ffff880298dfae98: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298dfae98: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880298dfae98: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 17953:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802772df0c0 x1710331746466560/t0(0) o106->lustre-OST0001@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 17953:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000404:0x52:0x0]// may get corrupted (rc -108)
LustreError: 19981:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298dfae98: namespace resource [0x160:0x0:0x0].0x0 (ffff88025d2851c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 19981:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298dfae98: namespace resource [0x300000401:0xc6:0x0].0x0 (ffff880263e8ca40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 19981:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 128 previous similar messages
Lustre: lustre-OST0001-osc-ffff880298dfae98: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: 16886:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880264490cc0 x1710331746747136/t0(0) o41->lustre-MDT0002-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp-pre-0-2.0'
Lustre: 16526:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 506 < left 5170, rollback = 7
Lustre: 16526:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1966 previous similar messages
Lustre: 16526:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 120/480/0, destroy: 0/0/0
Lustre: 16526:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1967 previous similar messages
Lustre: 16526:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 122/15/0
Lustre: 16526:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1967 previous similar messages
Lustre: 16526:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 601/5170/0, punch: 0/0/0, quota 10/42/5
Lustre: 16526:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1967 previous similar messages
Lustre: 16526:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 120/2040/0, delete: 0/0/0
Lustre: 16526:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1967 previous similar messages
Lustre: 16526:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16526:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1967 previous similar messages
Lustre: 17721:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802754bd180 x1710331746874112/t0(0) o106->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
INFO: task mv:16414 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880284bca140 11344 16414 15893 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:16890 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880294a490c0 11344 16890 16548 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
Lustre: 16553:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 511 < left 32213, rollback = 9
LustreError: 17308:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x83:0x0]: rc = -2
LustreError: 20270:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x200000403:0x3:0x0] mdc close failed: rc = -13
Lustre: 16553:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 509 < left 5170, rollback = 7
Lustre: 16553:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 8028 previous similar messages
Lustre: 11084:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 502 < left 671, rollback = 2
Lustre: 11084:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 48 previous similar messages
Lustre: 18159:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880294a33838 x1710331747825600/t0(0) o105->LOV_OSC_UUID@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
Lustre: 17140:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/10, destroy: 0/0/0
Lustre: 17140:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 20516 previous similar messages
Lustre: 17140:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/671/0
Lustre: 17140:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 20516 previous similar messages
Lustre: 17140:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 17140:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 20516 previous similar messages
Lustre: 17140:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 17140:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 20516 previous similar messages
Lustre: 17140:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 17140:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 20516 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff880284ddb880/0x4105371653f93cb2 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x4105371653f93c8f expref: 3389 pid: 17420 timeout: 553 lvb_type: 0
LustreError: 10985:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631099883 with bad export cookie 4685211556386458677
LustreError: 12673:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88008b881300 x1710331749400576/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 12673:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
Lustre: lustre-OST0001-osc-ffff880298df53d8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880298df53d8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 10985:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 2 previous similar messages
Lustre: 4143:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0xbb:0x0]// may get corrupted (rc -108)
LustreError: 22055:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298df53d8: namespace resource [0x300000401:0x247:0x0].0x0 (ffff880268b44540) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 504 < left 21080, rollback = 7
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 21317 previous similar messages
Lustre: 17508:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 512 < left 8346, rollback = 9
Lustre: 17508:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 42 previous similar messages
Lustre: lustre-OST0001: haven't heard from client 1154ae0e-996a-42a4-bcbf-d44ba44315ac (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880264b22e98, cur 1631099930 expire 1631099900 last 1631099883
INFO: task mv:18505 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88028921d2c0 11168 18505 16726 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 22055:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298df53d8: namespace resource [0x300000400:0x130:0x0].0x0 (ffff88007b3874c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
LustreError: 11-0: lustre-OST0001-osc-ffff880298df53d8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298df53d8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880298df53d8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 11103:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '1' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 1' to finish migration.
LustreError: 11238:0:(out_handler.c:910:out_tx_end()) lustre-MDT0000-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
LustreError: 12289:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 12289:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 12289:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 12289:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 510 < left 7377, rollback = 9
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 34 previous similar messages
Lustre: 17103:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 378/1512/0, destroy: 0/0/0
Lustre: 17103:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 25573 previous similar messages
Lustre: 17103:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 380/15/0
Lustre: 17103:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 25573 previous similar messages
Lustre: 17103:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1891/16264/0, punch: 0/0/0, quota 10/106/7
Lustre: 17103:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 25573 previous similar messages
Lustre: 17103:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 378/6426/0, delete: 0/0/0
Lustre: 17103:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 25573 previous similar messages
Lustre: 17103:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17103:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 25573 previous similar messages
Lustre: 4133:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c5a2de00 x1710331754505344/t0(0) o10->lustre-OST0002-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100043 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
LustreError: 16861:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x8b:0x0]: rc = -2
Lustre: 18804:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 16307, rollback = 7
Lustre: 18804:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 25338 previous similar messages
Lustre: dir [0x200000404:0x131:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x200000404:0x131:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
INFO: task mv:22344 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88028d4d5c40 11344 22344 16062 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:22403 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88008dc7a980 11216 22403 15874 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:22804 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880286c26340 11216 22804 16052 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 11092:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration.
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880260bfde00/0x410537165407acea lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x2a4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->917503) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x410537165407ac26 expref: 2318 pid: 20724 timeout: 741 lvb_type: 0
LustreError: 28212:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88028cc05d28 ns: filter-lustre-OST0003_UUID lock: ffff88007227e580/0x41053716540fbbac lrc: 3/0,0 mode: PR/PR res: [0x380000400:0x45a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000000000 nid: 0@lo remote: 0x41053716540fab3d expref: 2326 pid: 28212 timeout: 0 lvb_type: 1
LustreError: 10996:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800803570c0 x1710331757550720/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 10996:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 43 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 4 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880298dfae98: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0xe4:0x0]// may get corrupted (rc -108)
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000404:0x1f7:0x0]// may get corrupted (rc -108)
LustreError: 28288:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298dfae98: namespace resource [0x380000400:0x430:0x0].0x0 (ffff880277b0ccc0) refcount nonzero (2) after lock cleanup; forcing cleanup.
LustreError: 28288:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 306 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 16997:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff88024b928cc0 x1710331758206208/t0(0) o104->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
LustreError: 12289:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 12289:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 21203:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 499 < left 2140, rollback = 2
Lustre: 21203:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 44 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 114s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff8802790a8f40/0x4105371654093a93 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x361:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->458751) gid 0 flags: 0x60000480000020 nid: 0@lo remote: 0x4105371654093a62 expref: 4143 pid: 12670 timeout: 800 lvb_type: 0
LustreError: 10997:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880075610680 x1710331759888704/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 10997:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 13 previous similar messages
LustreError: 19399:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631100146 with bad export cookie 4685211556386456563
LustreError: 11-0: lustre-OST0001-osc-ffff880298df53d8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298df53d8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880298df53d8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x11c:0x0]/ may get corrupted (rc -5)
Lustre: 19571:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff880085f570c0 x1710331759943296/t0(0) o400->lustre-MDT0001-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'kworker/u32:8.0'
Lustre: 19571:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 3 previous similar messages
Lustre: 4143:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000404:0x21b:0x0]// may get corrupted (rc -108)
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000404:0x16e:0x0]// may get corrupted (rc -108)
Lustre: 4129:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000404:0x22e:0x0]// may get corrupted (rc -108)
Lustre: 4129:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0xa1:0x0]// may get corrupted (rc -108)
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x15e:0x0]// may get corrupted (rc -108)
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000404:0x14b:0x0]/ may get corrupted (rc -108)
Lustre: 4131:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880248be0680 x1710331760077376/t0(0) o10->lustre-OST0001-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100237 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
LustreError: 18328:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880261541f80 x1710331760080064/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 18328:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 24 previous similar messages
Lustre: 17863:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff880249a9ac00 x1710331760073088/t0(0) o35->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'ls.0'
Lustre: 17863:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 5 previous similar messages
LustreError: 29623:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298df53d8: namespace resource [0x470:0x0:0x0].0x0 (ffff880077794040) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 29623:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 513 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
INFO: task mv:23853 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88026296c9d0 11344 23853 15935 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:24055 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88029412b1c0 11040 24055 15858 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:25295 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88026283c9d0 12128 25295 16464 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:26235 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880289418880 11344 26235 16872 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
Lustre: 4130:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88024c032c00 x1710331760763520/t0(0) o10->lustre-OST0003-osc-ffff880298df53d8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100236 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
Lustre: 19573:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff8802404789a8 x1710331760432448/t0(0) o106->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
Lustre: dir [0x280000404:0x8f:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: dir [0x280000404:0x113:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 18267:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 51/204/0, destroy: 1/4/0
Lustre: 18267:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 76147 previous similar messages
Lustre: 18267:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 55/148/0
Lustre: 18267:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 76147 previous similar messages
Lustre: 18267:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 256/2203/0, punch: 0/0/0, quota 8/56/0
Lustre: 18267:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 76147 previous similar messages
Lustre: 18267:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 52/883/0, delete: 2/5/1
Lustre: 18267:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 76146 previous similar messages
Lustre: 18267:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 18267:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 76147 previous similar messages
Lustre: 4146:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880265aeb240 x1710331763673280/t0(0) o10->lustre-OST0002-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100345 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88027d2a07c0/0x410537165414c239 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x2f6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000480010020 nid: 0@lo remote: 0x410537165414c232 expref: 2978 pid: 19295 timeout: 941 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 18497:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88024cdbf700 x1710331763697920/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 18497:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 35 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0003-osc-ffff880298dfae98: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
Lustre: 17204:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8800b2272ee8 x1710331763702784/t0(0) o35->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'cp.0'
LustreError: 31371:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298dfae98: namespace resource [0x380000400:0xc00:0x0].0x0 (ffff88026423de40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31371:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 379 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 16553:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 509 < left 63263, rollback = 7
Lustre: 16553:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 77134 previous similar messages
Lustre: 4146:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880265aeb240 x1710331765476160/t0(0) o10->lustre-OST0002-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100399 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 23904:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000403:0x1c4:0x0]: rc = -2
dd (1023) used greatest stack depth: 10080 bytes left
Lustre: 4138:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88024c7970c0 x1710331769150080/t0(0) o10->lustre-OST0001-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100481 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 135s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88007ba5ed00/0x410537165418ca18 lrc: 3/0,0 mode: PW/PW res: [0x425:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x410537165418c9fc expref: 2888 pid: 23239 timeout: 1051 lvb_type: 0
LustreError: 10997:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88026f80b880 x1710331769256384/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 10997:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 28 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff880298df53d8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff880298df53d8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff880298df53d8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 10991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631100417 with bad export cookie 4685211556388297248
LustreError: 10991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x310:0x0]/ may get corrupted (rc -108)
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x1fc:0x0]/ may get corrupted (rc -108)
LustreError: 5:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 1154ae0e-996a-42a4-bcbf-d44ba44315ac/ffff88028da86678 has 61440 pending on destroyed export
LustreError: 5:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 168046016 != fo_tot_granted 168107456
LustreError: 5:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 90112 != fo_tot_pending 151552
LustreError: 12688:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 168046016 != fo_tot_granted 168107456
LustreError: 12688:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 90112 != fo_tot_pending 151552
Lustre: 4131:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x1f3:0x0]// may get corrupted (rc -108)
Lustre: 4132:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x1da:0x0]// may get corrupted (rc -108)
Lustre: 4132:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000404:0x258:0x0]// may get corrupted (rc -108)
LustreError: 12691:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 168017344 != fo_tot_granted 168078784
LustreError: 12691:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
Lustre: 26609:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 501 < left 1027, rollback = 2
Lustre: 26609:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 177 previous similar messages
LustreError: 12689:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 168017344 != fo_tot_granted 168078784
LustreError: 12689:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 12689:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12689:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
Lustre: dir [0x200000404:0x174:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 9 previous similar messages
LustreError: 12691:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 176455104 != fo_tot_granted 176516544
LustreError: 12691:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 12691:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12691:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 1721:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298df53d8: namespace resource [0x430:0x0:0x0].0x0 (ffff88026552e5c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 1721:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 439 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
LustreError: 12689:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 176453440 != fo_tot_granted 176514880
LustreError: 12689:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 3 previous similar messages
LustreError: 12689:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12689:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 3 previous similar messages
Lustre: 4146:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880265aeb240 x1710331770190720/t0(0) o10->lustre-OST0002-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100529 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 4146:0:(client.c:1485:after_reply()) Skipped 2 previous similar messages
LustreError: 12682:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 192977216 != fo_tot_granted 193038656
LustreError: 12682:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 6 previous similar messages
LustreError: 12682:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12682:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 6 previous similar messages
LustreError: 12682:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 225691968 != fo_tot_granted 225753408
LustreError: 12682:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 9 previous similar messages
LustreError: 12682:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12682:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 9 previous similar messages
LustreError: 19055:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '9' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 9' to finish migration.
Lustre: ll_ost02_010: service thread pid 19978 was inactive for 40.050 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 22083, comm: ll_ost02_013 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22097, comm: ll_ost02_017 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 2 previous similar messages
Pid: 19978, comm: ll_ost02_010 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost02_004: service thread pid 17944 was inactive for 40.596 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: ll_ost04_013: service thread pid 19310 was inactive for 42.033 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
LustreError: 12679:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 267418176 != fo_tot_granted 267479616
LustreError: 12679:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 26 previous similar messages
LustreError: 12679:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12679:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 26 previous similar messages
Lustre: 4791:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (22/1s); client may timeout req@ffff880248caa5c0 x1710331774081472/t0(0) o101->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:92/0 lens 328/400 e 0 to 0 dl 1631100582 ref 2 fl Complete:/0/0 rc 301/301 job:'ln.0'
Lustre: 4791:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 18 previous similar messages
LustreError: 4791:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 22/1s ago req@ffff88028ab53ec0 x1710331774083776/t0(0) o101->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:92/0 lens 328/0 e 0 to 0 dl 1631100582 ref 2 fl Interpret:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 21987:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88027e2af0c0 x1710331774073728/t0(0) o1->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:92/0 lens 440/0 e 0 to 0 dl 1631100582 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 21987:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 17 previous similar messages
Lustre: 17834:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (22/1s); client may timeout req@ffff88007c3f6440 x1710331774073216/t0(0) o1->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:92/0 lens 440/432 e 0 to 0 dl 1631100582 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 17834:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 9 previous similar messages
Lustre: 4742:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880279ca6440 x1710331774072576/t0(0) o1->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:93/0 lens 440/432 e 0 to 0 dl 1631100583 ref 2 fl Interpret:/0/0 rc 0/0 job:'lfs.0'
Lustre: 4742:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 58 previous similar messages
Lustre: 22067:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (22/3s); client may timeout req@ffff88007236ac00 x1710331774071936/t0(0) o1->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:92/0 lens 440/432 e 0 to 0 dl 1631100582 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 22067:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 7 previous similar messages
Lustre: 19248:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88029954bec0 x1710331774338432/t0(0) o101->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:95/0 lens 328/0 e 0 to 0 dl 1631100585 ref 2 fl New:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 19248:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages
Lustre: ll_ost01_024: service thread pid 30499 was inactive for 64.091 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 21741:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 23/4s ago req@ffff880263b30680 x1710331774323840/t0(0) o101->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:95/0 lens 328/0 e 0 to 0 dl 1631100585 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'mkdir.0'
Lustre: 4948:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (23/4s); client may timeout req@ffff880241deb240 x1710331774326144/t0(0) o101->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:95/0 lens 328/0 e 0 to 0 dl 1631100585 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'mkdir.0'
Lustre: 4948:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 4 previous similar messages
LustreError: 21741:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 25 previous similar messages
Lustre: 11092:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x335:0x0] with magic=0xbd60bd0
Lustre: 18416:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000404:0x353:0x0] with magic=0xbd60bd0
Lustre: 18416:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: ll_ost05_023: service thread pid 18331 was inactive for 90.111 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 12 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8800768aed00/0x41053716541f92ab lrc: 3/0,0 mode: PW/PW res: [0x42a:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x41053716541f92a4 expref: 4841 pid: 12669 timeout: 1294 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 11-0: lustre-OST0003-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 21159:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631100622 with bad export cookie 4685211556388583989
LustreError: 21159:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0003-osc-ffff880298dfae98: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 21473:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802739ab880 x1710331776859264/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 21473:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Lustre: 4135:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631100562/real 1631100562] req@ffff880266367700 x1710331774323840/t0(0) o101->lustre-OST0001-osc-ffff880298df53d8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1631100610 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'mkdir.0'
Lustre: lustre-OST0001: Client 1154ae0e-996a-42a4-bcbf-d44ba44315ac (at 0@lo) reconnecting
Lustre: lustre-OST0001-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 1540:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli bc1e05b8-368a-4968-928a-1b45107d748e/ffff8802c4c40008 has 61440 pending on destroyed export
Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound).
Lustre: 18416:0:(service.c:1612:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=4, svcEst=58, delay=5452ms
Lustre: 18416:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88027832de00 x1710331773971904/t0(0) o36->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:144/0 lens 688/2888 e 0 to 0 dl 1631100634 ref 2 fl Interpret:/0/0 rc 0/0 job:'lfs.0'
Lustre: 18416:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 18 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0002_UUID lock: ffff8802420234c0/0x410537165425671e lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x117:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x41053716542566ca expref: 179 pid: 23972 timeout: 1328 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 3 previous similar messages
LustreError: 16626:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88008fa412a8 ns: mdt-lustre-MDT0002_UUID lock: ffff8802499bc3c0/0x4105371654263566 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 12 type: IBT gid 0 flags: 0x50200400000020 nid: 0@lo remote: 0x410537165425b361 expref: 24 pid: 16626 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-MDT0002-mdc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
LustreError: 1771:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x280000404:0x25a:0x0] mdc close failed: rc = -108
LustreError: 4753:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -5
LustreError: 32123:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0002-mdc-ffff880298dfae98: [0x280000403:0xc5:0x0] lock enqueue fails: rc = -108
Lustre: dir [0x240000403:0x2cf:0x0] stripe 1 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 3730:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x280000404:0x391:0x0] error -108.
LustreError: 3730:0:(vvp_io.c:1793:vvp_io_init()) Skipped 1 previous similar message
LustreError: 4134:0:(mdc_locks.c:1386:mdc_intent_getattr_async_interpret()) lustre-MDT0002-mdc-ffff880298dfae98: ldlm_cli_enqueue_fini() failed: rc = -108
LustreError: 5386:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x280000404:0x14a:0x0] mdc close failed: rc = -108
LustreError: 5386:0:(file.c:234:ll_close_inode_openhandle()) Skipped 10 previous similar messages
LustreError: 4498:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x389:0x0] error: rc = -108
LustreError: 4498:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 33 previous similar messages
Lustre: 31384:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631100557/real 1631100557] req@ffff880278329f80 x1710331773971904/t0(0) o36->lustre-MDT0001-mdc-ffff880298dfae98@0@lo:12/10 lens 688/4592 e 0 to 1 dl 1631100661 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 31384:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-MDT0001: Client bc1e05b8-368a-4968-928a-1b45107d748e (at 0@lo) reconnecting
Lustre: Skipped 3 previous similar messages
LustreError: 12679:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 264363328 != fo_tot_granted 264424768
LustreError: 12679:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 107 previous similar messages
LustreError: 12679:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 122880
LustreError: 12679:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 107 previous similar messages
LustreError: 21407:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration.
LustreError: 6434:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x200000404:0x349:0x0] mdc close failed: rc = -13
LustreError: 6434:0:(file.c:234:ll_close_inode_openhandle()) Skipped 8 previous similar messages
Lustre: 4139:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88024334de00 x1710331780871040/t0(0) o10->lustre-OST0003-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631100774 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 4139:0:(client.c:1485:after_reply()) Skipped 2 previous similar messages
Lustre: 28277:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x25e:0x0] with magic=0xbd60bd0
Lustre: 28277:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 11100:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x24b:0x0] with magic=0xbd60bd0
Lustre: 11100:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 7019:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298df53d8: inode [0x200000404:0x355:0x0] mdc close failed: rc = -13
Lustre: 23972:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x284:0x0] with magic=0xbd60bd0
Lustre: 23972:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 28263:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0
Lustre: 28263:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 132789 previous similar messages
Lustre: 28263:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 16/1472/0
Lustre: 28263:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 132789 previous similar messages
Lustre: 28263:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 28263:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 132788 previous similar messages
Lustre: 28263:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/3, delete: 0/0/0
Lustre: 28263:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 132790 previous similar messages
Lustre: 28263:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 28263:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 132789 previous similar messages
LustreError: 11123:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0xd4:0x0]: rc = -2
Lustre: 23904:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 485 < left 68355, rollback = 7
Lustre: 23904:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 137446 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff88024468e1c0/0x41053716542c1aa3 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x7bd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x41053716542c19d8 expref: 1931 pid: 2936 timeout: 1455 lvb_type: 0
LustreError: 25828:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631100833 with bad export cookie 4685211556387025348
LustreError: 25828:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 5 previous similar messages
LustreError: 11-0: lustre-OST0001-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298dfae98: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 9 previous similar messages
LustreError: 167-0: lustre-OST0001-osc-ffff880298dfae98: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
LustreError: 9214:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880298dfae98: namespace resource [0x7a0:0x0:0x0].0x0 (ffff880242858a40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 9214:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 117 previous similar messages
Lustre: 4133:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000404:0x416:0x0]/ may get corrupted (rc -108)
LustreError: 5:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli bc1e05b8-368a-4968-928a-1b45107d748e/ffff88027a65a548 has 28672 pending on destroyed export
LustreError: 12530:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli bc1e05b8-368a-4968-928a-1b45107d748e/ffff88028f60a548 has 61440 pending on destroyed export
LustreError: 8289:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x200000403:0x2c6:0x0] mdc close failed: rc = -108
LustreError: 7504:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -108
LustreError: 7504:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 1 previous similar message
Lustre: mdt04_002: service thread pid 11092 was inactive for 96.175 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
LustreError: 9625:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -108
LustreError: 9625:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 7 previous similar messages
LustreError: 9758:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 16859:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631100863 with bad export cookie 4685211556389030134
LustreError: 16859:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 10 previous similar messages
LustreError: 9892:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 9945:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -108
LustreError: 9945:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 38 previous similar messages
LustreError: 8547:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -108
LustreError: 10437:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x1:0x0] error: rc = -108
LustreError: 10437:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 115 previous similar messages
LustreError: 9605:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x200000404:0x2fe:0x0] mdc close failed: rc = -108
LustreError: 9605:0:(file.c:234:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 154:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 1154ae0e-996a-42a4-bcbf-d44ba44315ac/ffff8802649bc138 has 3649536 pending on destroyed export
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x50c:0x0]/ may get corrupted (rc -108)
Lustre: dir [0x280000405:0x89:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
Lustre: 12680:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff88028a60bec0 x1710331787360768/t0(0) o13->lustre-MDT0002-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp-pre-3-2.0'
Lustre: 12680:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
LustreError: 11909:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11909:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 11909:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11909:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 12 previous similar messages
LustreError: 31503:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 69142208 != fo_tot_granted 69265088
LustreError: 31503:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 259 previous similar messages
LustreError: 31503:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 1792 != fo_tot_pending 124672
LustreError: 31503:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 259 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880278a67840/0x410537165434fb8f lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x1c09:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->134217727) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x410537165434fb88 expref: 2335 pid: 30616 timeout: 1632 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 12 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: 28293:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff8802459d8008 ns: filter-lustre-OST0002_UUID lock: ffff88027f852d40/0x41053716543ae40d lrc: 2/0,0 mode: --/PR res: [0x340000400:0x2311:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x41053716543a327b expref: -99 pid: 28293 timeout: 0 lvb_type: 0
LustreError: 167-0: lustre-OST0002-osc-ffff880298dfae98: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 5 previous similar messages
LustreError: 30398:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff8802459d8008 ns: filter-lustre-OST0002_UUID lock: ffff8802624eed00/0x41053716543b2456 lrc: 2/0,0 mode: --/PR res: [0x340000400:0x2312:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x41053716543a3297 expref: -99 pid: 30398 timeout: 0 lvb_type: 0
LustreError: 30398:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) Skipped 2 previous similar messages
LustreError: 12670:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff8802459d8008 ns: filter-lustre-OST0002_UUID lock: ffff88025d3cb4c0/0x41053716543b38d8 lrc: 2/0,0 mode: --/PR res: [0x340000400:0x1222:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x41053716543a4e2e expref: -99 pid: 12670 timeout: 0 lvb_type: 0
LustreError: 12670:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) Skipped 56 previous similar messages
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000404:0x3b3:0x0]/ may get corrupted (rc -108)
Lustre: 11094:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 506 < left 983, rollback = 2
Lustre: 11094:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 965 previous similar messages
Lustre: 4138:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88027a61b240 x1710331791027008/t0(0) o10->lustre-OST0001-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631101057 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
LustreError: 26609:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '17' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 17' to finish migration.
LustreError: 11684:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff880298dfae98: namespace resource [0x340000400:0x2280:0x0].0x0 (ffff880267026d40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 11684:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 241 previous similar messages
LustreError: 11654:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880261e10cc0 x1710331793944064/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11654:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 231 previous similar messages
LustreError: 16662:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631101082 with bad export cookie 4685211556390509220
Lustre: 19511:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802768ff018 x1710331793925376/t0(0) o104->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
Lustre: 19511:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x47a:0x0]// may get corrupted (rc -108)
Lustre: 18424:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (51/1s); client may timeout req@ffff88026eafbec0 x1710331793409984/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:641/0 lens 440/432 e 1 to 0 dl 1631101131 ref 2 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 18424:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 18 previous similar messages
Lustre: 4903:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-3s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8800750057c0 x1710331793409536/t0(0) o1->bc1e05b8-368a-4968-928a-1b45107d748e@0@lo:641/0 lens 440/432 e 1 to 0 dl 1631101131 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: lustre-OST0003: haven't heard from client 1154ae0e-996a-42a4-bcbf-d44ba44315ac (at 0@lo) in 54 seconds. I think it's dead, and I am evicting it. exp ffff880077584a88, cur 1631101137 expire 1631101107 last 1631101083
LustreError: 30510:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802f1ecc138 ns: filter-lustre-OST0001_UUID lock: ffff88027937c000/0x4105371654447cb2 lrc: 3/0,0 mode: PW/PW res: [0x300000401:0x1338:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x4105371654447c3b expref: 4394 pid: 30510 timeout: 0 lvb_type: 0
LustreError: 20090:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631101149 with bad export cookie 4685211556390399481
LustreError: 20090:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 3 previous similar messages
LustreError: 30510:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 7 previous similar messages
Lustre: 4132:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x337:0x0]/ may get corrupted (rc -108)
Lustre: 4131:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x49b:0x0]/ may get corrupted (rc -108)
LustreError: 20234:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '2' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 2' to finish migration.
LustreError: 18380:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
Lustre: 4143:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88024ddd4500 x1710331800174208/t0(0) o10->lustre-OST0001-osc-ffff880298dfae98@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631101341 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 4143:0:(client.c:1485:after_reply()) Skipped 5 previous similar messages
LustreError: 28215:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
LustreError: 17446:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '0' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 0' to finish migration.
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff8802643a6940/0x410537165447ebb0 lrc: 3/0,0 mode: PW/PW res: [0x804:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0x410537165447eb9b expref: 4238 pid: 4796 timeout: 1982 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 4 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff880298dfae98: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 25 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff880298dfae98: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 5 previous similar messages
Lustre: 4129:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0xa0:0x0]/ may get corrupted (rc -108)
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000405:0x1b8:0x0]// may get corrupted (rc -108)
LustreError: 28215:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
LustreError: 15803:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff880298dfae98: namespace resource [0x810:0x0:0x0].0x0 (ffff880266ab96c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 15803:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 1914 previous similar messages
LustreError: 16030:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16030:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16346:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 12 previous similar messages
LustreError: 16346:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16346:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 16406:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16406:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16406:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
Lustre: lustre-OST0002: haven't heard from client bc1e05b8-368a-4968-928a-1b45107d748e (at 0@lo) in 49 seconds. I think it's dead, and I am evicting it. exp ffff88026725efc8, cur 1631101362 expire 1631101332 last 1631101313
LustreError: 16737:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16737:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16737:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 28195:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
LustreError: 16406:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16406:0:(lov_object.c:1299:lov_layout_change()) Skipped 5 previous similar messages
LustreError: 16406:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16406:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 5 previous similar messages
LustreError: 16406:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 16406:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 5 previous similar messages
LustreError: 18286:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '2' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 2' to finish migration.
LustreError: 17508:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
LustreError: 16860:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16860:0:(lov_object.c:1299:lov_layout_change()) Skipped 10 previous similar messages
LustreError: 16860:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16860:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 10 previous similar messages
LustreError: 16860:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 16860:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 10 previous similar messages
Lustre: 18367:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0
Lustre: 18367:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 295798 previous similar messages
Lustre: 18367:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 21/1917/0
Lustre: 18367:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 295801 previous similar messages
Lustre: 18367:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 18367:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 295802 previous similar messages
Lustre: 18367:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/5, delete: 0/0/0
Lustre: 18367:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 295799 previous similar messages
Lustre: 18367:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 18367:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 295794 previous similar messages
LustreError: 16882:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 16882:0:(lov_object.c:1299:lov_layout_change()) Skipped 7 previous similar messages
LustreError: 16882:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 16882:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 7 previous similar messages
LustreError: 16882:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 16882:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 7 previous similar messages
LustreError: 18804:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x8f:0x0]: rc = -2
Lustre: 28262:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 493 < left 81007, rollback = 7
Lustre: 28262:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 290292 previous similar messages
LustreError: 16861:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0xa7:0x0]: rc = -2
LustreError: 17307:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0xa7:0x0]: rc = -2
LustreError: 17325:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298dfae98: inode [0x240000404:0x483:0x0] mdc close failed: rc = -108
LustreError: 17325:0:(file.c:234:ll_close_inode_openhandle()) Skipped 15 previous similar messages
LustreError: 17913:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 17913:0:(lov_object.c:1299:lov_layout_change()) Skipped 13 previous similar messages
LustreError: 17913:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 17913:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 13 previous similar messages
LustreError: 17913:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 17913:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 13 previous similar messages
7[18271]: segfault at 8 ip 00007f8f193f27e8 sp 00007ffd45f6a010 error 4 in ld-2.17.so[7f8f193e7000+22000]
LustreError: 11085:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '12' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 12' to finish migration.
LustreError: 18680:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631101451 with bad export cookie 4685211556391565618
Lustre: 4129:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x645:0x0]// may get corrupted (rc -108)
LustreError: 18086:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x280000403:0x645:0x0] error -108.
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x497:0x0]/ may get corrupted (rc -108)
LustreError: 16620:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli 1154ae0e-996a-42a4-bcbf-d44ba44315ac/ffff88008ee48958 has 122880 pending on destroyed export
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x52f:0x0]/ may get corrupted (rc -108)
Lustre: 4146:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000405:0x1dd:0x0]/ may get corrupted (rc -108)
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000404:0x4bf:0x0]/ may get corrupted (rc -108)
Lustre: 4146:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x4b8:0x0]// may get corrupted (rc -108)
LustreError: 11122:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0xd1:0x0]: rc = -2
LustreError: 12687:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 64783680 != fo_tot_granted 64935232
LustreError: 12687:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 540 previous similar messages
LustreError: 12687:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 151552
LustreError: 12687:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 540 previous similar messages
Lustre: 23904:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 8s req@ffff88007bcc4b40 x1710331806824832/t0(0) o35->1154ae0e-996a-42a4-bcbf-d44ba44315ac@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'rm.0'
Lustre: 23904:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 11 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 11 previous similar messages
LustreError: 18615:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298dfae98: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 18615:0:(lov_object.c:1299:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 18615:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 18615:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 18615:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 18615:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: 28263:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000405:0xce:0x0] with magic=0xbd60bd0
Lustre: 28263:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000405:0x62:0x0]// may get corrupted (rc -108)
Lustre: 11099:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 511 < left 25668, rollback = 9
Lustre: 11099:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 291 previous similar messages
LustreError: 11082:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '10' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 10' to finish migration.
LustreError: 22057:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298dfae98: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 22057:0:(lov_object.c:1299:lov_layout_change()) Skipped 34 previous similar messages
LustreError: 22057:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x398:0x0]: rc = -22
LustreError: 22057:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 34 previous similar messages
LustreError: 22057:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 22057:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 34 previous similar messages
ptlrpc_watchdog_fire: 30 callbacks suppressed
Lustre: ll_ost01_043: service thread pid 18829 was inactive for 40.095 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 18829, comm: ll_ost01_043 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x6e1:0x0]/ may get corrupted (rc -108)
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000404:0x47e:0x0]/ may get corrupted (rc -108)
LustreError: 5:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 1154ae0e-996a-42a4-bcbf-d44ba44315ac/ffff88008a34ae98 has 28672 pending on destroyed export
Lustre: 4139:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x737:0x0]/ may get corrupted (rc -5)
LustreError: 17946:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88022e33c500 x1710331817452928/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 17946:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 763 previous similar messages
Lustre: 4134:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0x233:0x0]/ may get corrupted (rc -108)
Lustre: 4133:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0x1db:0x0]/ may get corrupted (rc -108)
Lustre: 4133:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000403:0x4d3:0x0]/ may get corrupted (rc -108)
13[24250]: segfault at 8 ip 00007f5f9ee9f7e8 sp 00007ffc0afc5150 error 4 in ld-2.17.so[7f5f9ee94000+22000]
Lustre: dir [0x280000403:0x608:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
Lustre: 4138:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88024b97f700 x1710331819612864/t0(0) o10->lustre-OST0003-osc-ffff880298df53d8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631101888 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4138:0:(client.c:1485:after_reply()) Skipped 5 previous similar messages
Lustre: ll_ost02_031: service thread pid 4793 was inactive for 40.055 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 4793, comm: ll_ost02_031 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
17[25070]: segfault at 8 ip 00007f54b9a227e8 sp 00007ffdc2ed5850 error 4 in ld-2.17.so[7f54b9a17000+22000]
LustreError: 26211:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x200000405:0x26d:0x0] error -22.
LustreError: 25173:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000405:0x26d:0x0]: rc = -22
LustreError: 25173:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 57 previous similar messages
LustreError: 25173:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 25173:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 55 previous similar messages
LustreError: 26468:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880298df53d8: cannot apply new layout on [0x200000403:0x398:0x0] : rc = -22
LustreError: 26468:0:(lov_object.c:1299:lov_layout_change()) Skipped 59 previous similar messages
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88025f628400/0x4105371654604b34 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x358a:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->1310719) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x41053716546048b7 expref: 2155 pid: 8445 timeout: 2594 lvb_type: 0
LustreError: 10998:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 9 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880298df53d8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 15 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff880298df53d8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 10 previous similar messages
Lustre: 4135:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x63e:0x0]/ may get corrupted (rc -108)
Lustre: 4136:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0x1f4:0x0]// may get corrupted (rc -108)
Lustre: 4135:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x4b8:0x0]/ may get corrupted (rc -108)
LustreError: 27577:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298df53d8: namespace resource [0x380000400:0x3cc0:0x0].0x0 (ffff8802521796c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 27577:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 3035 previous similar messages
LustreError: 19337:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '12' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 12' to finish migration.
LustreError: 19337:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) Skipped 2 previous similar messages
LustreError: 28219:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298df53d8: inode [0x280000403:0x830:0x0] mdc close failed: rc = -13
LustreError: 28219:0:(file.c:234:ll_close_inode_openhandle()) Skipped 20 previous similar messages
Lustre: 16920:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 16920:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 172516 previous similar messages
Lustre: 16920:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 16920:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 172516 previous similar messages
Lustre: 16920:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0
Lustre: 16920:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 172519 previous similar messages
Lustre: 16920:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/2, delete: 1/1/0
Lustre: 16920:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 172517 previous similar messages
Lustre: 16920:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 16920:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 172517 previous similar messages
Lustre: 10383:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 510 < left 781, rollback = 7
Lustre: 10383:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 172982 previous similar messages
Lustre: 18367:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x621:0x0] with magic=0xbd60bd0
Lustre: 18367:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: ll_ost07_054: service thread pid 11676 was inactive for 100.075 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 11676, comm: ll_ost07_054 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 28419:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x668:0x0] with magic=0xbd60bd0
Lustre: 28419:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 10991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631102033 with bad export cookie 4685211556392595059
LustreError: 10991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 3 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298dfae98: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 10 previous similar messages
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000403:0x690:0x0]// may get corrupted (rc -108)
Lustre: 4143:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x280000403:0x855:0x0]// may get corrupted (rc -108)
Lustre: 4143:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x240000405:0x228:0x0]// may get corrupted (rc -108)
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0x31c:0x0]// may get corrupted (rc -108)
LustreError: 12685:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 261520064 != fo_tot_granted 261548736
LustreError: 12685:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 696 previous similar messages
LustreError: 12685:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 4218880 != fo_tot_pending 4247552
LustreError: 12685:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 696 previous similar messages
19[32206]: segfault at 0 ip (null) sp 00007ffcca91a548 error 14 in 14[400000+6000]
LustreError: 32222:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x200000403:0x398:0x0] error -22.
Lustre: 4143:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.185@tcp:/lustre/fid: [0x200000405:0x3cd:0x0]/ may get corrupted (rc -5)
LustreError: 375:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 1154ae0e-996a-42a4-bcbf-d44ba44315ac/ffff8802c4cb6fc8 has 1196032 pending on destroyed export
Lustre: lustre-OST0002-osc-ffff880298df53d8: Connection restored to 192.168.123.185@tcp (at 0@lo)
Lustre: Skipped 8 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 10 PID: 19337 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel drm_panel_orientation_quirks virtio_blk libata serio_raw i2c_core floppy
CPU: 10 PID: 19337 Comm: mdt05_009 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0c14262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0bf433d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0bf4437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cb8e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04de6f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa049d6b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa048a760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa04904ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04875ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04b8a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa082160d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa07f1892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0ef3f3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0dc8408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0da9b2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04dd375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0daab86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0e7c0d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0e3be53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0e3c087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0e1155c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0e1e657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07e026e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032d45e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa078b990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa078d529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa078c950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 28bda789721aa800 ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-1): ldiskfs_getblk:888: inode #193: block 20973: comm mdt05_009: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-1-8.
LDISKFS-fs (dm-1): Remounting filesystem read-only
LustreError: 19337:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0001: error reading offset 659456 (block 161, size 48, offs 659448), credits 0/0: rc = -28
LustreError: 19337:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -28
LustreError: 19337:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 19337:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0001: can't update reply_data file: rc = -30
LustreError: 19337:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0001: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-1) in osd_trans_stop:2082: error 28
LustreError: 12911:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0000-osc-MDT0001: fail to cancel 1 llog-records: rc = -30

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa179fe52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 80000000a5c56067 PUD a5c57067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm crct10dif_pclmul drm crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 4 PID: 3140 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8802c4199280 ti: ffff8800a368c000 task.ti: ffff8800a368c000
RIP: 0010:[<ffffffffa179fe52>] [<ffffffffa179fe52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff8800a368fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802a0ab55e8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000247 RDI: 0000000000000247
RBP: ffff8800a368fc48 R08: ffff88008af77660 R09: 0000000000000000
R10: ffff88008af77640 R11: ffff8802d9c2c5a8 R12: 0000000000000000
R13: ffff8802e6e79738 R14: ffff880063e83898 R15: 0000000000000030
FS: 00007f1952ccc740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 00000000a045c000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa178d561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff811b515b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e2e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1cf>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff810dbb8b>] ? put_prev_entity+0x8b/0x400
[<ffffffff810d5979>] ? pick_next_entity+0xa9/0x190
LustreError: 9080:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 9080:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 3 previous similar messages
[<ffffffff810d7e1c>] ? set_next_entity+0x3c/0xe0
[<ffffffff811ecf6d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81245a38>] ? vfs_write+0x178/0x230
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 531, rollback = 7
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/0
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 446/531/0, punch: 0/0/0, quota 6/6/0
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 511 < left 530, rollback = 7
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/0
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 446/531/1, punch: 0/0/0, quota 6/6/0
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 500 < left 528, rollback = 7
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 436 previous similar messages
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 436 previous similar messages
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/9
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 436 previous similar messages
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 446/531/3, punch: 0/0/0, quota 6/6/0
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 436 previous similar messages
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 436 previous similar messages
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 436 previous similar messages
Lustre: 22308:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 508 < left 610, rollback = 2
Lustre: 22308:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0
Lustre: 22308:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1745 previous similar messages
Lustre: 22308:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 8/610/0
Lustre: 22308:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1745 previous similar messages
Lustre: 22308:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0
Lustre: 22308:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1745 previous similar messages
Lustre: 22308:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/3, delete: 1/1/0
Lustre: 22308:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1745 previous similar messages
Lustre: 22308:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 22308:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1745 previous similar messages
Lustre: 21526:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 509 < left 7406, rollback = 7
Lustre: 21526:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1745 previous similar messages
Lustre: 24409:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 582, rollback = 2
Lustre: 24409:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
LustreError: 21560:0:(out_handler.c:910:out_tx_end()) lustre-MDT0001-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
LustreError: 21606:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 21606:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
Lustre: 23621:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 1491, rollback = 7
Lustre: 23621:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 688 previous similar messages
Lustre: 23621:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0
Lustre: 23621:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 700 previous similar messages
Lustre: 23621:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 23621:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 700 previous similar messages
Lustre: 23621:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1482/1491/0, punch: 0/0/0, quota 6/6/0
Lustre: 23621:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 700 previous similar messages
Lustre: 23621:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0
Lustre: 23621:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 700 previous similar messages
Lustre: 23621:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 23621:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 700 previous similar messages
LustreError: 25302:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880071ab1bf8: inode [0x280000403:0x1b:0x0] mdc close failed: rc = -13
Lustre: 25571:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 595, rollback = 7
Lustre: 25571:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1481 previous similar messages
Lustre: 25571:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 25571:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1481 previous similar messages
Lustre: 25571:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/0
Lustre: 25571:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1481 previous similar messages
Lustre: 25571:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 510/595/0, punch: 0/0/0, quota 6/6/0
Lustre: 25571:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1481 previous similar messages
Lustre: 25571:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 25571:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1481 previous similar messages
Lustre: 25571:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 25571:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1481 previous similar messages
LustreError: 25553:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 15' to finish migration.
Lustre: 24830:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 8533, rollback = 9
Lustre: 24830:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
LustreError: 26545:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880071ab1bf8: inode [0x200000404:0x6:0x0] mdc close failed: rc = -13
Lustre: 25536:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 501 < left 610, rollback = 2
LustreError: 26897:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880071ab1bf8: inode [0x200000404:0x6:0x0] mdc close failed: rc = -13
Lustre: 21544:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 510 < left 61457, rollback = 7
Lustre: 21544:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 12155 previous similar messages
Lustre: 21544:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1429/5716/0, destroy: 1/4/1
Lustre: 21544:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 12169 previous similar messages
Lustre: 21544:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1429/0/0
Lustre: 21544:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 12169 previous similar messages
Lustre: 21544:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 7146/61457/0, punch: 0/0/0, quota 6/6/0
Lustre: 21544:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 12169 previous similar messages
Lustre: 21544:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1429/24293/0, delete: 2/5/1
Lustre: 21544:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 12169 previous similar messages
Lustre: 21544:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0
Lustre: 21544:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 12169 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 14 PID: 21544 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm crct10dif_pclmul drm crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 14 PID: 21544 Comm: mdt_rdpg07_000 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0d6b4ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0d5c9eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0d5f417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0d7faa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0d7fad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0c9bd08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04de59d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04e2cf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0efbaf4>] mdt_mfd_close+0x1f4/0x870 [mdt]
[<ffffffffa0f01e14>] mdt_close_internal+0xb4/0x240 [mdt]
[<ffffffffa0f02224>] mdt_close+0x284/0x850 [mdt]
[<ffffffffa07e52d4>] ? tgt_request_preprocess.isra.27+0xf4/0x850 [ptlrpc]
[<ffffffffa07e626e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa03ca45e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0791990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa0793529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0792950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 331e0184efa6e1eb ]---
Lustre: 21524:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 509 < left 7445, rollback = 9
Lustre: 21524:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 12 previous similar messages
Lustre: 24236:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 504 < left 36904, rollback = 7
Lustre: 24236:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 20392 previous similar messages
Lustre: 24236:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 858/3432/0, destroy: 0/0/0
Lustre: 24236:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 20393 previous similar messages
Lustre: 24236:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 860/15/0
Lustre: 24236:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 20393 previous similar messages
Lustre: 24236:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 4291/36904/0, punch: 0/0/0, quota 14/174/7
Lustre: 24236:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 20392 previous similar messages
Lustre: 24236:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 858/14586/0, delete: 0/0/0
Lustre: 24236:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 20392 previous similar messages
Lustre: 24236:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24236:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 20393 previous similar messages
LustreError: 25330:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 15' to finish migration.
LustreError: 26115:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '18' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 18' to finish migration.
Lustre: 21525:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 510 < left 13497, rollback = 9
LustreError: 28322:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x200000404:0x6:0x0] mdc close failed: rc = -13
Lustre: 23621:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 511 < left 2973, rollback = 9
Lustre: 23621:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88021d029300/0x1765494a08b78931 lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x2:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->131071) gid 0 flags: 0x60000480000020 nid: 0@lo remote: 0x1765494a08b78907 expref: 1542 pid: 22185 timeout: 3654 lvb_type: 0
LustreError: 29091:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802a7093880 x1710325411333568/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0003-osc-ffff880071ab1bf8: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880071ab1bf8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 18309:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000404:0x6e:0x0]// may get corrupted (rc -108)
LustreError: 29571:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880071ab1bf8: namespace resource [0x10:0x0:0x0].0x0 (ffff88024b4f3240) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 510 < left 15350, rollback = 9
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 902/3608/0, destroy: 1/4/0
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 17200 previous similar messages
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 906/148/0
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 17200 previous similar messages
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 4511/38796/0, punch: 0/0/0, quota 10/74/0
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 17201 previous similar messages
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 903/15350/0, delete: 2/5/1
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 17201 previous similar messages
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 17201 previous similar messages
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 509 < left 38796, rollback = 7
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 17197 previous similar messages
LustreError: 21606:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 21606:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: mdt04_006: service thread pid 24743 was inactive for 62.053 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 24743, comm: mdt04_006 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_reint_object_lock+0x2c/0x60 [mdt]
[<0>] mdt_reint_striped_lock+0x89/0x5b0 [mdt]
[<0>] mdt_migrate_parent_lock+0x3f/0xb0 [mdt]
[<0>] mdt_reint_migrate+0x8b7/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 22693, comm: mdt00_003 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
Lustre: mdt05_003: service thread pid 23873 was inactive for 62.091 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 23873, comm: mdt05_003 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_rename+0x31c/0x2ff0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
Lustre: mdt06_004: service thread pid 24550 was inactive for 62.230 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt03_006: service thread pid 25330 was inactive for 62.158 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt02_010: service thread pid 25571 was inactive for 62.125 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt02_004: service thread pid 24451 was inactive for 62.276 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: mdt00_009: service thread pid 24869 was inactive for 62.119 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff8802b01b8400/0x1765494a08ba3f65 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x20c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x1765494a08ba3efc expref: 1481 pid: 26582 timeout: 3732 lvb_type: 0
LustreError: 30361:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802c9deb240 x1710325414736320/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 22203:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880278215d28 ns: filter-lustre-OST0000_UUID lock: ffff88022074c780/0x1765494a08c35df3 lrc: 3/0,0 mode: --/PW res: [0x2c0000400:0x6ab:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x1765494a08c35dec expref: 1271 pid: 22203 timeout: 0 lvb_type: 0
LustreError: 21494:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631093716 with bad export cookie 1685834217842756103
Lustre: lustre-OST0000-osc-ffff88008c3a92a8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0000-osc-ffff88008c3a92a8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: 32507:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff88008c3a92a8: namespace resource [0x2c0000400:0x70e:0x0].0x0 (ffff880097738cc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 32507:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 41 previous similar messages
Lustre: lustre-OST0000-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: mdt03_003: service thread pid 23007 was inactive for 62.264 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff880092dc4f00/0x1765494a08be38c6 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x14:0x0].0x0 bits 0x12/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x1765494a08be389c expref: 80 pid: 25553 timeout: 3753 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff880071ab1bf8: operation mds_reint to node 0@lo failed: rc = -107
Lustre: lustre-MDT0001-mdc-ffff880071ab1bf8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff880071ab1bf8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 26665:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880071ab1bf8: inode [0x240000403:0x4c:0x0] mdc close failed: rc = -108
LustreError: 28358:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -108
Lustre: lustre-MDT0001-mdc-ffff880071ab1bf8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: 24550:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 498 < left 1055, rollback = 2
Lustre: 24550:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2 previous similar messages
LustreError: 25553:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '18' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 18' to finish migration.
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880226d7f480/0x1765494a08c07323 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x2f5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x1765494a08c072cf expref: 1703 pid: 22200 timeout: 3781 lvb_type: 0
LustreError: 22299:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880092434b40 x1710325416144384/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0003-osc-ffff880071ab1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880071ab1bf8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 18311:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x240000404:0xe0:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: 18315:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802b29cd7c0 x1710325416918976/t0(0) o10->lustre-OST0003-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631093831 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: dir [0x280000403:0x17:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 18316:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802b1d16440 x1710325417333696/t0(0) o10->lustre-OST0001-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631093806 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 25055:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/3, destroy: 0/0/0
Lustre: 25055:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 32341 previous similar messages
Lustre: 25055:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/671/0
Lustre: 25055:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 32341 previous similar messages
Lustre: 25055:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0
Lustre: 25055:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 32341 previous similar messages
Lustre: 25055:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 25055:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 32341 previous similar messages
Lustre: 25055:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 25055:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 32341 previous similar messages
Lustre: 21517:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 3321, rollback = 7
Lustre: 21517:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 32275 previous similar messages
Lustre: dir [0x240000404:0xda:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 18315:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802b29cd7c0 x1710325420127232/t0(0) o10->lustre-OST0003-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631093888 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 18315:0:(client.c:1485:after_reply()) Skipped 74 previous similar messages
Lustre: 18315:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802a7038040 x1710325420400000/t0(0) o10->lustre-OST0002-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631093902 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 18316:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802b1d16440 x1710325420720448/t0(0) o10->lustre-OST0001-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631093919 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
LustreError: 2715:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x200000404:0x5f:0x0] mdc close failed: rc = -13
LustreError: 2715:0:(file.c:234:ll_close_inode_openhandle()) Skipped 8 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff8802baf41e40/0x1765494a08c543fc lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x1e9:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x1765494a08c543f5 expref: 2064 pid: 26572 timeout: 3883 lvb_type: 0
LustreError: 23266:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b67a6440 x1710325421128320/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 23266:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 27 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88008c3a92a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff88008c3a92a8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 18309:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x240000404:0x1b9:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: 26713:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 504 < left 1294, rollback = 2
Lustre: 26713:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 72 previous similar messages
LustreError: 21539:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x1fa:0x0]: rc = -2
Lustre: 27359:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880225b0cb40 x1710325422217728/t0(0) o13->lustre-MDT0002-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp-pre-0-2.0'
Lustre: 22206:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802b02c9f80 x1710325422565824/t0(0) o101->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'truncate.0'
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880094f2ed00/0x1765494a08ce0e0c lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x53c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x1765494a08ce0e05 expref: 3389 pid: 26563 timeout: 4012 lvb_type: 0
LustreError: 6722:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88009f844b40 x1710325426848384/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 6722:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 24 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880071ab1bf8: operation ost_getattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880071ab1bf8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 18315:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000403:0x26c:0x0]/ may get corrupted (rc -108)
Lustre: 18316:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000403:0x27e:0x0]/ may get corrupted (rc -108)
Lustre: 18320:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802d1859940 x1710325427022144/t0(0) o10->lustre-OST0003-osc-ffff88008c3a92a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094063 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
Lustre: 18320:0:(client.c:1485:after_reply()) Skipped 27 previous similar messages
LustreError: 7391:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880071ab1bf8: namespace resource [0x380000401:0x53f:0x0].0x0 (ffff880092303c40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 7391:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 5 previous similar messages
Lustre: 29080:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x26a:0x0] with magic=0xbd60bd0
LustreError: 3625:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '8' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 8' to finish migration.
Lustre: 21528:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x26f:0x0] with magic=0xbd60bd0
Lustre: 21528:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 6721:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8802b9f3c500 x1710325425372672/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:317/0 lens 440/0 e 0 to 0 dl 1631094012 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 22694:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x276:0x0] with magic=0xbd60bd0
Lustre: 6708:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88009f900680 x1710325425403200/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:318/0 lens 440/0 e 0 to 0 dl 1631094013 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 6708:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 237 previous similar messages
Lustre: 22694:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 28146:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-2s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8802232b9940 x1710325425645440/t0(0) o2->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:319/0 lens 440/0 e 0 to 0 dl 1631094014 ref 2 fl New:/0/ffffffff rc 0/-1 job:'chown.0'
Lustre: 28146:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 335 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff8800935007c0/0x1765494a08cd796f lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x718:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x1765494a08cd795a expref: 3429 pid: 32326 timeout: 4033 lvb_type: 0
LustreError: 25344:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d0ca1300 x1710325427962368/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 26200:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631094017 with bad export cookie 1685834217842756110
Lustre: lustre-OST0001-osc-ffff88008c3a92a8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff88008c3a92a8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 25344:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 114 previous similar messages
LustreError: 25344:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802782153d8 ns: filter-lustre-OST0001_UUID lock: ffff8802c4cf2d40/0x1765494a08d4be42 lrc: 3/0,0 mode: --/PW res: [0x300000400:0x783:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x1765494a08d4be2d expref: 3336 pid: 25344 timeout: 0 lvb_type: 0
LustreError: 6690:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/6s ago req@ffff88022460b240 x1710325425354112/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:317/0 lens 440/0 e 0 to 0 dl 1631094012 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 6688:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/6s); client may timeout req@ffff8800944e9f80 x1710325425354240/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:317/0 lens 440/0 e 0 to 0 dl 1631094012 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 6690:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 65 previous similar messages
Lustre: 6705:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88029ea970c0 x1710325425938048/t0(0) o2->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:323/0 lens 440/0 e 0 to 0 dl 1631094018 ref 2 fl New:/0/ffffffff rc 0/-1 job:'chown.0'
Lustre: 6705:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 772 previous similar messages
LustreError: 6690:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/7s ago req@ffff8802cfd26440 x1710325425393472/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:318/0 lens 440/0 e 0 to 0 dl 1631094013 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 6690:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 170 previous similar messages
Lustre: 6690:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/7s); client may timeout req@ffff8802cfd26440 x1710325425393472/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:318/0 lens 440/0 e 0 to 0 dl 1631094013 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 6690:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 257 previous similar messages
LustreError: 28866:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631094021 with bad export cookie 1685834217842755879
LustreError: 28866:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 2 previous similar messages
LustreError: 6680:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/8s ago req@ffff88009f9e8040 x1710325425601024/t0(0) o2->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:319/0 lens 440/0 e 0 to 0 dl 1631094014 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chown.0'
Lustre: 6730:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/8s); client may timeout req@ffff88009f867700 x1710325425601152/t0(0) o2->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:319/0 lens 440/0 e 0 to 0 dl 1631094014 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chown.0'
Lustre: 6730:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 315 previous similar messages
LustreError: 6680:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 372 previous similar messages
LustreError: 30106:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/2s ago req@ffff880093779f80 x1710325426100672/t0(0) o2->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:327/0 lens 440/0 e 0 to 0 dl 1631094022 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chmod.0'
LustreError: 30106:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 1697 previous similar messages
Lustre: 30106:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/2s); client may timeout req@ffff880093779f80 x1710325426100672/t0(0) o2->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:327/0 lens 440/0 e 0 to 0 dl 1631094022 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chmod.0'
Lustre: 30106:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 1737 previous similar messages
LustreError: 5563:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 16015095-046a-4c50-a5f2-954fef3e91d0/ffff8802650dc138 has 28672 pending on destroyed export
LustreError: 5563:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 268069696 != fo_tot_granted 268098368
LustreError: 5563:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 896 != fo_tot_pending 29568
LustreError: 7391:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880071ab1bf8: namespace resource [0x380000401:0x5df:0x0].0x0 (ffff8802ceb474c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 7391:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 758 previous similar messages
Lustre: lustre-OST0003-osc-ffff880071ab1bf8: Connection restored to 192.168.123.215@tcp (at 0@lo)
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 268067904 != fo_tot_granted 268096576
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
Lustre: 18307:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x240000405:0xb9:0x0]/ may get corrupted (rc -108)
Lustre: 18315:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x249:0x0]/ may get corrupted (rc -108)
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 268067904 != fo_tot_granted 268096576
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 7782:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff88008c3a92a8: namespace resource [0x2c0000400:0x770:0x0].0x0 (ffff880090035940) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 7782:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 228 previous similar messages
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 268067904 != fo_tot_granted 268096576
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
Lustre: 18310:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c0143880 x1710325428705920/t0(0) o10->lustre-OST0001-osc-ffff88008c3a92a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094085 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 22211:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 271242304 != fo_tot_granted 271270976
LustreError: 22211:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 22211:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22211:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 272688192 != fo_tot_granted 272716864
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 2998272 != fo_tot_pending 3026944
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
Lustre: 21510:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 503 < left 85580, rollback = 7
Lustre: 21510:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 98391 previous similar messages
Lustre: 21510:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1990/7960/0, destroy: 0/0/0
Lustre: 21510:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 98572 previous similar messages
Lustre: 21510:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 1992/15/0
Lustre: 21510:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 98574 previous similar messages
Lustre: 21510:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 9951/85580/0, punch: 0/0/0, quota 14/174/8
Lustre: 21510:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 98573 previous similar messages
Lustre: 21510:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1990/33830/0, delete: 0/0/0
Lustre: 21510:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 98572 previous similar messages
Lustre: 21510:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21510:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 98571 previous similar messages
Lustre: 18318:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631093975/real 1631093975] req@ffff8802b8a97700 x1710325425912768/t0(0) o2->lustre-OST0000-osc-ffff880071ab1bf8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1631094038 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'chown.0'
Lustre: lustre-OST0000-osc-ffff880071ab1bf8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0002: Client lustre-MDT0002-mdtlov_UUID (at 0@lo) reconnecting
Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0001: deleting orphan objects from 0x0:2693 to 0x0:2721
Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:2642 to 0x340000400:2657
LustreError: 3122:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 269689920 != fo_tot_granted 269718592
LustreError: 3122:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 4 previous similar messages
LustreError: 3122:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 3122:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 4 previous similar messages
Lustre: lustre-OST0003: deleting orphan objects from 0x380000400:2674 to 0x380000400:2689
Lustre: 18310:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c0143880 x1710325431196800/t0(0) o10->lustre-OST0001-osc-ffff88008c3a92a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094152 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 271322688 != fo_tot_granted 271351360
LustreError: 22212:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 11 previous similar messages
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22212:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 11 previous similar messages
Lustre: 26563:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631094068/real 1631094068] req@ffff88009f9fb880 x1710325429779136/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 1 dl 1631094118 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:''
Lustre: 26563:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 422 previous similar messages
Lustre: 18314:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631094069/real 1631094069] req@ffff8802c1732c00 x1710325429217856/t0(0) o1->lustre-OST0002-osc-ffff88008c3a92a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1631094113 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'lfs.0'
Lustre: 18314:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0002: Client 1c8284fc-3353-43bb-bb2e-6d88a2482b3d (at 0@lo) reconnecting
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 6 previous similar messages
Lustre: 18314:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631094071/real 1631094071] req@ffff880214da8680 x1710325429290624/t0(0) o1->lustre-OST0003-osc-ffff88008c3a92a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1631094123 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'lfs.0'
Lustre: 18314:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 94 previous similar messages
Lustre: lustre-OST0003: Client 1c8284fc-3353-43bb-bb2e-6d88a2482b3d (at 0@lo) reconnecting
Lustre: 24334:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 500 < left 521, rollback = 2
Lustre: 24334:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 181 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88009ee28040/0x1765494a08d7dbc4 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x937:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0x1765494a08d7dbbd expref: 4017 pid: 6099 timeout: 4182 lvb_type: 0
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 21497:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802cf79bec0 x1710325434611520/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0002-osc-ffff880071ab1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff880071ab1bf8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 8 previous similar messages
LustreError: 21497:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 27 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff880071ab1bf8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
LustreError: 22217:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 271028032 != fo_tot_granted 271056704
LustreError: 22217:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 21 previous similar messages
LustreError: 22217:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22217:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 21 previous similar messages
LustreError: 9871:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff880071ab1bf8: namespace resource [0x340000401:0x5cf:0x0].0x0 (ffff8802be16a840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 9871:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 2003 previous similar messages
Lustre: 7748:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631094112/real 1631094112] req@ffff880090a39f80 x1710325432446464/t0(0) o101->lustre-MDT0001-mdc-ffff880071ab1bf8@0@lo:12/10 lens 576/40248 e 0 to 1 dl 1631094181 ref 2 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'file_exec.sh.0'
Lustre: 7748:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
Lustre: lustre-MDT0001: Client 16015095-046a-4c50-a5f2-954fef3e91d0 (at 0@lo) reconnecting
Lustre: 18318:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802bbbe6a80 x1710325436037632/t0(0) o10->lustre-OST0001-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094270 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: dir [0x240000404:0x27c:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff8802ae296d00/0x1765494a08dafa73 lrc: 4/0,0 mode: PR/PR res: [0x240000404:0x207:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x1765494a08dafa3b expref: 112 pid: 24725 timeout: 4269 lvb_type: 0
LustreError: 26566:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88009a406fc8 ns: mdt-lustre-MDT0001_UUID lock: ffff8802bb3dcf00/0x1765494a08de6da0 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1765494a08de6d92 expref: 5 pid: 26566 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff880071ab1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 167-0: lustre-MDT0001-mdc-ffff880071ab1bf8: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 10290:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000403:0x1:0x0] error: rc = -5
LustreError: 13299:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880071ab1bf8: inode [0x240000405:0x125:0x0] mdc close failed: rc = -108
LustreError: 9134:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0001-mdc-ffff880071ab1bf8: [0x240000403:0x1:0x0] lock enqueue fails: rc = -108
Lustre: lustre-MDT0001-mdc-ffff880071ab1bf8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 3 previous similar messages
LustreError: 30361:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880090a3a5c0 x1710325439721664/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 30361:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 15 previous similar messages
Lustre: 18316:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000404:0x345:0x0]// may get corrupted (rc -5)
LustreError: 13471:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88008c3a92a8: namespace resource [0xdd5:0x0:0x0].0x0 (ffff8802a1faa5c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 13471:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 119 previous similar messages
Lustre: 8784:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802b1d3c500 x1710325439936256/t0(0) o101->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'cp.0'
LustreError: 22210:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 26671808 != fo_tot_granted 26700480
LustreError: 22210:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 51 previous similar messages
LustreError: 22210:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 22210:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 51 previous similar messages
ptlrpc_watchdog_fire: 10 callbacks suppressed
Lustre: mdt01_008: service thread pid 29080 was inactive for 92.264 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 29080, comm: mdt01_008 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0xaca/0x2590 [mdt]
[<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0xaa6/0x16d0 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt04_009: service thread pid 24813 was inactive for 89.250 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 24813, comm: mdt04_009 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 11-0: lustre-MDT0002-mdc-ffff88008c3a92a8: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-MDT0002-mdc-ffff88008c3a92a8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
LustreError: 9815:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x280000404:0x317:0x0] mdc close failed: rc = -108
LustreError: 9815:0:(file.c:234:ll_close_inode_openhandle()) Skipped 6 previous similar messages
LustreError: 11043:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -108
LustreError: 11462:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x1:0x0] error: rc = -108
LustreError: 12692:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0002-mdc-ffff88008c3a92a8: [0x280000403:0x2a5:0x0] lock enqueue fails: rc = -108
LustreError: 12692:0:(mdc_request.c:1436:mdc_read_page()) Skipped 9 previous similar messages
Lustre: dir [0x280000403:0x2a0:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x240000405:0x75:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x200000403:0x240:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 5 previous similar messages
Lustre: 26713:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x247:0x0] with magic=0xbd60bd0
Lustre: 26713:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 2085:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800975325c0 x1710325443205248/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 2085:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8802aa90f840/0x1765494a08e53610 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0xae3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x1765494a08e53609 expref: 3414 pid: 2085 timeout: 4397 lvb_type: 0
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 3 previous similar messages
Lustre: 22265:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x269:0x0] with magic=0xbd60bd0
Lustre: 22265:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0003-osc-ffff880071ab1bf8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 3 previous similar messages
Lustre: 3625:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802c3c470c0 x1710325443262208/t0(0) o101->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:0/0 lens 4616/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'touch.0'
Lustre: 18314:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x211:0x0]/ may get corrupted (rc -108)
Lustre: 18314:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000404:0x3b8:0x0]/ may get corrupted (rc -108)
LustreError: 5563:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 1c8284fc-3353-43bb-bb2e-6d88a2482b3d/ffff8802a60f53d8 has 28672 pending on destroyed export
Lustre: 21556:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802ceb16440 x1710325443375616/t0(0) o101->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'mdt04_012.0'
Lustre: 21556:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
Lustre: 22265:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x26d:0x0] with magic=0xbd60bd0
Lustre: 22265:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 16315:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88008c3a92a8: namespace resource [0x340000401:0x83f:0x0].0x0 (ffff8802b13ffc40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 16315:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 3 previous similar messages
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 2 previous similar messages
Lustre: 27721:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0000-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x200000403:0x1cf:0x0] with magic=0xbd60bd0
Lustre: 27721:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: 18315:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802aed470c0 x1710325444388032/t0(0) o10->lustre-OST0003-osc-ffff880071ab1bf8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094433 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 18315:0:(client.c:1485:after_reply()) Skipped 3 previous similar messages
Lustre: 30255:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880296e28040 x1710325444385024/t0(0) o106->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
LustreError: 8361:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802b0593880 x1710325447537408/t0(0) o106->lustre-OST0000@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 21477:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631094507 with bad export cookie 1685834217844686836
LustreError: 21477:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 2 previous similar messages
LustreError: 8361:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 38 previous similar messages
LustreError: 11-0: lustre-OST0000-osc-ffff88008c3a92a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 3 previous similar messages
LustreError: 18626:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff88008c3a92a8: namespace resource [0x2c0000401:0x81f:0x0].0x0 (ffff8802bba01bc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 18626:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 45 previous similar messages
LustreError: 2078:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 21/12s ago req@ffff8802c9f592f8 x1710325447484416/t0(0) o101->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:77/0 lens 328/0 e 0 to 0 dl 1631094527 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chmod.0'
Lustre: 2078:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (21/12s); client may timeout req@ffff8802c9f592f8 x1710325447484416/t0(0) o101->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:77/0 lens 328/0 e 0 to 0 dl 1631094527 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'chmod.0'
LustreError: 22221:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 236193088 != fo_tot_granted 236221760
LustreError: 22221:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 135 previous similar messages
LustreError: 22221:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22221:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 135 previous similar messages
Lustre: 25087:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 476/1904/0, destroy: 1/4/0
Lustre: 25087:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 166891 previous similar messages
Lustre: 25087:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 484/283/0
Lustre: 25087:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 166891 previous similar messages
Lustre: 25087:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 2381/20478/0, punch: 0/0/0, quota 6/6/0
Lustre: 25087:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 166890 previous similar messages
Lustre: 25087:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 478/8124/0, delete: 3/6/1
Lustre: 25087:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 166891 previous similar messages
Lustre: 25087:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 25087:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 166891 previous similar messages
Lustre: 25087:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 507 < left 20478, rollback = 7
Lustre: 25087:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 166700 previous similar messages
Lustre: lustre-OST0001-osc-ffff880071ab1bf8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
LustreError: 2265:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631094626 with bad export cookie 1685834217845267542
LustreError: 2265:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 4 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff880094466940/0x1765494a08f04129 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x365:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x1765494a08f0406c expref: 176 pid: 27738 timeout: 4678 lvb_type: 0
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 3 previous similar messages
LustreError: 11-0: lustre-MDT0000-mdc-ffff88008c3a92a8: operation mds_close to node 0@lo failed: rc = -107
LustreError: Skipped 20 previous similar messages
LustreError: 167-0: lustre-MDT0000-mdc-ffff88008c3a92a8: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: Skipped 3 previous similar messages
LustreError: 18481:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x200000404:0x4dc:0x0] mdc close failed: rc = -108
LustreError: 18481:0:(file.c:234:ll_close_inode_openhandle()) Skipped 17 previous similar messages
LustreError: 18675:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -108
LustreError: 20346:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 20334:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-MDT0000-mdc-ffff88008c3a92a8: namespace resource [0x200000404:0x4fd:0x0].0x0 (ffff8800935f82c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20334:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 421 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: 27790:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 496 < left 1027, rollback = 2
Lustre: 27790:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 184 previous similar messages
LustreError: 24777:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880290bd6678 ns: mdt-lustre-MDT0001_UUID lock: ffff8802d8a134c0/0x1765494a08f0f603 lrc: 3/0,0 mode: PR/PR res: [0x240000405:0x11f:0x0].0x0 bits 0x12/0x0 rrc: 11 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1765494a08f0e4a6 expref: 16 pid: 24777 timeout: 0 lvb_type: 0
LustreError: 16525:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000405:0x11f:0x0] error: rc = -5
LustreError: 19346:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 19847:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0001-mdc-ffff88008c3a92a8: [0x240000401:0xd:0x0] lock enqueue fails: rc = -108
LustreError: 19847:0:(mdc_request.c:1436:mdc_read_page()) Skipped 13 previous similar messages
Lustre: dir [0x200000404:0x4fd:0x0] stripe 1 readdir failed: -108, directory is partially accessed!
LustreError: 16525:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 61 previous similar messages
LustreError: 25087:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '19' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 19' to finish migration.
LustreError: 21683:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 33/1s ago req@ffff88008f17f018 x1710325455158848/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:286/0 lens 440/0 e 0 to 0 dl 1631094736 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 21679:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (33/1s); client may timeout req@ffff88022343f700 x1710325455159104/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:286/0 lens 440/0 e 0 to 0 dl 1631094736 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 21683:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 2 previous similar messages
Lustre: ll_ost01_017: service thread pid 32330 was inactive for 42.009 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 32330, comm: ll_ost01_017 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 8775, comm: ll_ost01_038 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 4074, comm: ll_ost01_029 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost01_048: service thread pid 20453 was inactive for 42.189 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 27175:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802cc60e440 x1710325458329984/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 27175:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 71 previous similar messages
Lustre: 18322:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x48e:0x0]/ may get corrupted (rc -108)
Lustre: 18321:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x240000406:0xe2:0x0]// may get corrupted (rc -108)
Lustre: 18321:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x3fe:0x0]// may get corrupted (rc -108)
Lustre: 27337:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (56/21s); client may timeout req@ffff88028b68e6c8 x1710325455036288/t0(0) o1->1c8284fc-3353-43bb-bb2e-6d88a2482b3d@0@lo:302/0 lens 440/432 e 1 to 0 dl 1631094752 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 27337:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 47 previous similar messages
Lustre: 27247:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8800a24dbec0 x1710325461693056/t0(0) o35->16015095-046a-4c50-a5f2-954fef3e91d0@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'touch.0'
LustreError: 32351:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802782137e8 ns: filter-lustre-OST0002_UUID lock: ffff8802bf174000/0x1765494a0901965e lrc: 3/0,0 mode: --/PW res: [0x340000400:0x12af:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0x1765494a09019657 expref: 6252 pid: 32351 timeout: 0 lvb_type: 0
LustreError: 32351:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 7 previous similar messages
Lustre: 18309:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x8f:0x0]// may get corrupted (rc -108)
Lustre: 18310:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000405:0x224:0x0]/ may get corrupted (rc -108)
Lustre: 18318:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c33bc500 x1710325462241408/t0(0) o10->lustre-OST0003-osc-ffff88008c3a92a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631094956 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 18318:0:(client.c:1485:after_reply()) Skipped 2 previous similar messages
Lustre: 9316:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 9s req@ffff8802b67a4500 x1710325463289280/t0(0) o103->lustre-MDT0001-mdtlov_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'jbd2/dm-1-8.0'
Lustre: dir [0x200000405:0x3c:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 21798:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 21798:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 21798:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 21798:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 27738:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0x212:0x0] with magic=0xbd60bd0
Lustre: 27738:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 19 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880071ab1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 21473:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631095030 with bad export cookie 1685834217847157346
LustreError: Skipped 263 previous similar messages
Lustre: 18316:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x55a:0x0]/ may get corrupted (rc -108)
ptlrpc_watchdog_fire: 89 callbacks suppressed
Lustre: mdt01_011: service thread pid 25124 was inactive for 106.263 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 25124, comm: mdt01_011 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa25/0xec0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x436/0xa30 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x1cc/0x510 [mdt]
[<0>] mdt_remote_object_lock+0x2a/0x30 [mdt]
[<0>] mdt_revoke_remote_lookup_lock+0x86/0x150 [mdt]
[<0>] mdt_link_parents_lock.isra.52+0x6b1/0x15f0 [mdt]
[<0>] mdt_reint_migrate+0x930/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 22213:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 153599936 != fo_tot_granted 153628608
LustreError: 22213:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 398 previous similar messages
LustreError: 22213:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 22213:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 398 previous similar messages
Lustre: mdt01_002: service thread pid 21511 was inactive for 106.245 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 21511, comm: mdt01_002 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 25155, comm: mdt01_013 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Lustre: mdt01_007: service thread pid 25087 was inactive for 106.150 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 88 previous similar messages
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0xaca/0x2590 [mdt]
[<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0xaa6/0x16d0 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 25879:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0002-mdc-ffff88008c3a92a8: [0x280000403:0x1:0x0] lock enqueue fails: rc = -108
LustreError: 25765:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x3bb:0x0] error: rc = -5
LustreError: 25765:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 8 previous similar messages
LustreError: 25879:0:(mdc_request.c:1436:mdc_read_page()) Skipped 20 previous similar messages
LustreError: 26721:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x280000405:0x1de:0x0] mdc close failed: rc = -108
LustreError: 26721:0:(file.c:234:ll_close_inode_openhandle()) Skipped 22 previous similar messages
LustreError: 26721:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-MDT0002-mdc-ffff88008c3a92a8: namespace resource [0x280000403:0x1:0x0].0x0 (ffff880099bda0c0) refcount nonzero (2) after lock cleanup; forcing cleanup.
LustreError: 26721:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 1315 previous similar messages
LustreError: 23079:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '2' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 2' to finish migration.
LustreError: 25168:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 3' to finish migration.
Lustre: 24813:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0x215:0x0] with magic=0xbd60bd0
Lustre: 24813:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 21519:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 87/348/0, destroy: 0/0/0
Lustre: 21519:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 213503 previous similar messages
Lustre: 21519:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 89/15/0
Lustre: 21519:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 213502 previous similar messages
Lustre: 21519:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 436/3751/0, punch: 0/0/0, quota 14/142/9
Lustre: 21519:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 213503 previous similar messages
Lustre: 21519:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 87/1479/0, delete: 0/0/0
Lustre: 21519:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 213502 previous similar messages
Lustre: 21519:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21519:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 213501 previous similar messages
Lustre: 21519:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 502 < left 3751, rollback = 7
Lustre: 21519:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 213280 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff8802d08dc3c0/0x1765494a0908a5bc lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x181a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0x1765494a0908a584 expref: 2562 pid: 25344 timeout: 5223 lvb_type: 0
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 7 previous similar messages
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 9 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff88008c3a92a8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 7 previous similar messages
Lustre: lustre-OST0002-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 7 previous similar messages
Lustre: 17176:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802b215a598 x1710325479499776/t0(0) o105->LOV_OSC_UUID@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
Lustre: 17176:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 14 previous similar messages
Lustre: 21513:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 496 < left 671, rollback = 2
Lustre: 21513:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 286 previous similar messages
LustreError: 6724:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d2ef5e00 x1710325481948672/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 6724:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 112 previous similar messages
Lustre: 18321:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000406:0xc1:0x0]/ may get corrupted (rc -108)
LustreError: 21510:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802c169b7e8 ns: mdt-lustre-MDT0001_UUID lock: ffff88009ef34000/0x1765494a09169419 lrc: 3/0,0 mode: PR/PR res: [0x240000406:0x54:0x0].0x0 bits 0x12/0x0 rrc: 19 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x1765494a0916936a expref: 16 pid: 21510 timeout: 0 lvb_type: 0
LustreError: 26179:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000406:0x54:0x0] error: rc = -5
LustreError: 26179:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
LustreError: 28565:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008c3a92a8: inode [0x240000406:0x54:0x0] mdc close failed: rc = -108
LustreError: 28565:0:(file.c:234:ll_close_inode_openhandle()) Skipped 10 previous similar messages
Lustre: 27721:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000405:0x2af:0x0] with magic=0xbd60bd0
Lustre: 27721:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 23 previous similar messages
Lustre: dir [0x200000403:0x506:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: 18322:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88029ea63880 x1710325486842496/t0(0) o10->lustre-OST0003-osc-ffff88008c3a92a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631095520 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 18322:0:(client.c:1485:after_reply()) Skipped 2 previous similar messages
LustreError: 30775:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631095452 with bad export cookie 1685834217845718538
LustreError: 16784:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 1c8284fc-3353-43bb-bb2e-6d88a2482b3d/ffff8802abe28958 has 28672 pending on destroyed export
Lustre: 18313:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000406:0x12b:0x0]// may get corrupted (rc -108)
Lustre: 18314:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x631:0x0]/ may get corrupted (rc -108)
Lustre: 18313:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x259:0x0]/ may get corrupted (rc -108)
Lustre: 18310:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000406:0xdb:0x0]// may get corrupted (rc -108)
ptlrpc_watchdog_fire: 2 callbacks suppressed
Lustre: ll_ost06_046: service thread pid 18714 was inactive for 40.032 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 18714, comm: ll_ost06_046 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 9645, comm: ll_ost06_024 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 9654, comm: ll_ost06_028 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost06_042: service thread pid 18693 was inactive for 40.933 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 25019:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000408:0x23:0x0]: rc = -2
LustreError: 2335:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631095542 with bad export cookie 1685834217849066841
LustreError: 2335:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff880071ab1bf8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 14 previous similar messages
Lustre: 25939:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff8802c82b8058 x1710325490877184/t0(0) o104->LOV_OSC_UUID@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
Lustre: 25939:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 1 previous similar message
Lustre: dir [0x240000408:0x19:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 1675:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli 16015095-046a-4c50-a5f2-954fef3e91d0/ffff8802a9b06678 has 28672 pending on destroyed export
Lustre: 18317:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000406:0xc8:0x0]/ may get corrupted (rc -108)
Lustre: 18318:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x240000406:0x352:0x0]/ may get corrupted (rc -108)
Lustre: 18312:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x1f6:0x0]// may get corrupted (rc -108)
LustreError: 1675:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 16015095-046a-4c50-a5f2-954fef3e91d0/ffff880292196fc8 has 28672 pending on destroyed export
LustreError: 25055:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '8' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 8' to finish migration.
LustreError: 22218:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 152056640 != fo_tot_granted 152085312
LustreError: 22218:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 552 previous similar messages
LustreError: 22218:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 22218:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 552 previous similar messages
LustreError: 21538:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000407:0x144:0x0]: rc = -2
LustreError: 1675:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 1c8284fc-3353-43bb-bb2e-6d88a2482b3d/ffff8802264fca88 has 61440 pending on destroyed export
Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Lustre: 26618:0:(service.c:1612:ptlrpc_at_check_timed()) earlyQ=0 reqQ=0 recA=1, svcEst=49, delay=5849ms
Lustre: 18321:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x1f6:0x0]// may get corrupted (rc -108)
Lustre: 18322:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x746:0x0]// may get corrupted (rc -108)
Lustre: 18321:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x316:0x0]/ may get corrupted (rc -108)
Lustre: 18322:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x72a:0x0]// may get corrupted (rc -108)
LustreError: 3545:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88008c3a92a8: namespace resource [0x380000400:0x1f20:0x0].0x0 (ffff8802daea56c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 3545:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 2914 previous similar messages
Lustre: ll_ost02_022: service thread pid 32735 was inactive for 98.020 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 25 previous similar messages
LustreError: 21484:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631095778 with bad export cookie 1685834217849732807
LustreError: 21484:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 4 previous similar messages
LustreError: 24917:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88028e5737e8 ns: mdt-lustre-MDT0000_UUID lock: ffff8802cf04c000/0x1765494a092edb40 lrc: 3/0,0 mode: CR/CR res: [0x200000405:0x2d7:0x0].0x0 bits 0xa/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200400000000 nid: 0@lo remote: 0x1765494a092edb1d expref: 195 pid: 24917 timeout: 0 lvb_type: 0
LustreError: 24917:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 6 previous similar messages
LustreError: 2693:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000403:0x741:0x0] error: rc = -5
LustreError: 4766:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 1158:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0000-mdc-ffff880071ab1bf8: [0x200000400:0x28:0x0] lock enqueue fails: rc = -108
LustreError: 1158:0:(mdc_request.c:1436:mdc_read_page()) Skipped 6 previous similar messages
LustreError: 2693:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 13 previous similar messages
LustreError: 4958:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 4958:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 3 previous similar messages
LustreError: 3039:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0000-mdc-ffff880071ab1bf8: [0x200000400:0x25:0x0] lock enqueue fails: rc = -108
LustreError: 23589:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x200000405:0x2d7:0x0] error -108.
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 509 < left 32776, rollback = 7
Lustre: 24261:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 291182 previous similar messages
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 762/3048/0, destroy: 1/4/0
Lustre: 24261:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 291582 previous similar messages
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 766/148/0
Lustre: 24261:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 291584 previous similar messages
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 3811/32776/0, punch: 0/0/0, quota 10/74/0
Lustre: 24261:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 291587 previous similar messages
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 763/12970/0, delete: 2/5/1
Lustre: 24261:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 291586 previous similar messages
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 24261:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 291582 previous similar messages
Lustre: dir [0x200000403:0x701:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff8802d5388040/0x1765494a0925e81e lrc: 3/0,0 mode: PW/PW res: [0x300000401:0x2eb6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->786431) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x1765494a0925e817 expref: 2502 pid: 25488 timeout: 5890 lvb_type: 0
LustreError: 21499:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 27 previous similar messages
Lustre: lustre-OST0001-osc-ffff88008c3a92a8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 12 previous similar messages
LustreError: 21535:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0x506:0x0]: rc = -2
LustreError: 167-0: lustre-OST0001-osc-ffff88008c3a92a8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: Skipped 12 previous similar messages
Lustre: 18307:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000403:0x708:0x0]// may get corrupted (rc -5)
LustreError: 1181:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli 1c8284fc-3353-43bb-bb2e-6d88a2482b3d/ffff8802c82ba548 has 57344 pending on destroyed export
LustreError: 27185:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 1c8284fc-3353-43bb-bb2e-6d88a2482b3d/ffff8802c1e14138 has 61440 pending on destroyed export
Lustre: 18320:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x200000405:0x386:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff88008c3a92a8: Connection restored to 192.168.123.215@tcp (at 0@lo)
Lustre: Skipped 13 previous similar messages
Lustre: 21506:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 501 < left 760, rollback = 2
Lustre: 21506:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 369 previous similar messages
LustreError: 25019:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000405:0x354:0x0]: rc = -2
LustreError: 5748:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631095929 with bad export cookie 1685834217849799601
LustreError: 5748:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 11 previous similar messages
Lustre: 18314:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.215@tcp:/lustre/fid: [0x280000403:0x72d:0x0]// may get corrupted (rc -108)
------------[ cut here ]------------
WARNING: CPU: 4 PID: 5678 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm crct10dif_pclmul drm crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 4 PID: 5678 Comm: mdt02_015 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0d9a262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0d7a33d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0d7a437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cc5e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04e46f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa04a36b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa0490760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa04964ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa048d5ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04bea39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa082760d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa07f7892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0fa8f3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0e7d408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0e5eb2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04e3375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0e5fb86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0f310d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0ef0e53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0ef1087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0ec655c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0ed3657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07e626e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa03ca45e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0791990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa0793529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffffa0792950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 331e0184efa6e1ec ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-1): ldiskfs_getblk:888: inode #176: block 53819: comm mdt02_015: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-1-8.
LDISKFS-fs (dm-1): Remounting filesystem read-only
LustreError: 5678:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0001: error reading offset 598016 (block 146, size 40, offs 598000), credits 0/0: rc = -28
LustreError: 5678:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -28
LustreError: 5678:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 5678:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0001: can't update reply_data file: rc = -30
LustreError: 5678:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0001: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-1) in osd_trans_stop:2082: error 28
LustreError: 5678:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0001: failed to stop transaction: rc = -28
LustreError: 5678:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0001-osd: stop trans failed: rc = -30
LustreError: 22736:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0001-osc-MDT0001: fail to cancel 36 llog-records: rc = -30
LustreError: 22736:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0001-osc-MDT0001: can't cancel 36 records: rc = -30
LustreError: 8491:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 21783:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff8800a10fa508 commit error: 2
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LDISKFS-fs error (device dm-1) in osd_trans_stop:2082: IO failure
LustreError: 8428:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8428:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 4 previous similar messages
LustreError: 22736:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0001-osc-MDT0001: fail to cancel 101 llog-records: rc = -30
LustreError: 22736:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 8 previous similar messages
LustreError: 22736:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0001-osc-MDT0001: can't cancel 101 records: rc = -30
LustreError: 22736:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 5 previous similar messages
LustreError: 22004:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 2 previous similar messages
LustreError: 22004:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 22004:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 23212:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0003-osc-MDT0001: can't cancel 6 records: rc = -30
LustreError: 24140:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 24140:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 5 previous similar messages
LustreError: 21606:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 22736:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0001-osc-MDT0001: can't cancel record: rc = -30
LustreError: 22004:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 22004:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 6 previous similar messages
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 2 previous similar messages
LustreError: 22728:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel 39 records: rc = -30
LustreError: 22728:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 2 previous similar messages
LustreError: 8817:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8817:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 22728:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0000-osc-MDT0001: fail to cancel 9 llog-records: rc = -30
LustreError: 22728:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 7 previous similar messages
LustreError: 22728:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel 9 records: rc = -30
LustreError: 22728:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel 270 records: rc = -30
LustreError: 22728:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 1 previous similar message
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 22004:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 6 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa103ee52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 800000027128c067 PUD 276898067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi crct10dif_pclmul crct10dif_common drm crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata i2c_core floppy
CPU: 3 PID: 6763 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff88028ea45c40 ti: ffff880251e08000 task.ti: ffff880251e08000
RIP: 0010:[<ffffffffa103ee52>] [<ffffffffa103ee52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff880251e0bbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880260e876b8 RCX: 0000000000000000
RDX: 0000000100006e64 RSI: ffff880328c0b5d8 RDI: ffff880328c0b5b8
RBP: ffff880251e0bc48 R08: ffff880241c498d8 R09: ffff880251e0ba60
R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
R13: ffff88027c33ed58 R14: ffff88029ab10a58 R15: 0000000000000030
FS: 00007f8d1682b740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000286312000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa102c561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff811b515b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e2e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1cf>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff811ecf6d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81245a38>] ? vfs_write+0x178/0x230
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: lustre-OST0000-osc-ffff8802998a8958: disconnect after 21s idle
Lustre: 11143:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 788, rollback = 2
Lustre: 11143:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/3, destroy: 0/0/0
Lustre: 11143:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 11143:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 11143:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/1, delete: 1/1/0
Lustre: 11143:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 11143:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 507 < left 788, rollback = 2
Lustre: 11143:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/3, destroy: 0/0/0
Lustre: 11143:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 11143:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 11143:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/2, delete: 1/1/0
Lustre: 11143:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1193, rollback = 7
Lustre: 11120:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 11120:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11120:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/0
Lustre: 11120:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11120:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1108/1193/0, punch: 0/0/0, quota 6/6/0
Lustre: 11120:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11120:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 11120:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 8 previous similar messages
LustreError: 11329:0:(out_handler.c:910:out_tx_end()) lustre-MDT0000-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
Lustre: 11120:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11120:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 8 previous similar messages
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1193, rollback = 7
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 481 < left 1171, rollback = 7
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2478 previous similar messages
Lustre: 11120:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/0, destroy: 0/0/0
Lustre: 11120:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 2695 previous similar messages
Lustre: 11120:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/192/9
Lustre: 11120:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 2695 previous similar messages
Lustre: 11120:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1108/1193/22, punch: 0/0/0, quota 6/6/0
Lustre: 11120:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 2695 previous similar messages
Lustre: 11120:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/34/0, delete: 0/0/0
Lustre: 11120:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 2696 previous similar messages
Lustre: 11120:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11120:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 2697 previous similar messages
Lustre: 11140:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 498 < left 671, rollback = 2
Lustre: 11140:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 17384:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 500 < left 47267, rollback = 7
Lustre: 17384:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3019 previous similar messages
Lustre: 11132:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 503 < left 877, rollback = 2
Lustre: 11132:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 11132:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0
Lustre: 11132:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 7217 previous similar messages
Lustre: 11132:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 11132:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 7217 previous similar messages
Lustre: 11132:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 11132:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 7217 previous similar messages
Lustre: 11132:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 11132:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 7217 previous similar messages
Lustre: 11132:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 11132:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 7217 previous similar messages
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 18660:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 505 < left 521, rollback = 2
Lustre: 18660:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 11128:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 511 < left 798, rollback = 7
Lustre: 11128:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 4405 previous similar messages
Lustre: 11128:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0
Lustre: 11128:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 11128:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 11128:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 11128:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 789/798/0, punch: 0/0/0, quota 8/80/0
Lustre: 11128:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 11128:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0
Lustre: 11128:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 20 previous similar messages
Lustre: 11128:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11128:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 582, rollback = 2
Lustre: 11120:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 18660:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 510 < left 798, rollback = 7
Lustre: 18660:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1563 previous similar messages
Lustre: dir [0x200000403:0x56:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 17393:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 511 < left 18699, rollback = 9
Lustre: 17393:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 17393:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1099/4396/0, destroy: 1/4/0
Lustre: 17393:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 5419 previous similar messages
Lustre: 17393:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 1103/148/0
Lustre: 17393:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 5419 previous similar messages
Lustre: 17393:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5496/47267/0, punch: 0/0/0, quota 10/74/0
Lustre: 17393:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 5419 previous similar messages
Lustre: 17393:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1100/18699/0, delete: 2/5/0
Lustre: 17393:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 5419 previous similar messages
Lustre: 17393:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 17393:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 5419 previous similar messages
Lustre: 17469:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1179, rollback = 7
Lustre: 17469:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3850 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 10 PID: 11141 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi crct10dif_pclmul crct10dif_common drm crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata i2c_core floppy
CPU: 10 PID: 11141 Comm: mdt05_001 Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0bf54ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0be69eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0be9417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0c09aa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0c09ad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0c9ed08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04e859d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04eccf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0e1e440>] mdt_object_put+0x30/0x110 [mdt]
[<ffffffffa0e25e18>] mdt_reint_unlink+0x918/0x19b0 [mdt]
[<ffffffffa050e79e>] ? lu_ucred+0x1e/0x30 [obdclass]
[<ffffffffa0e15af2>] ? ucred_set_audit_enabled.isra.13+0x22/0x60 [mdt]
[<ffffffffa0e2d087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0e0255c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0e0f657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07f026e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa033545e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa079b990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa079d529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa079c950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 647798641f8bde2e ]---
LustreError: 21041:0:(dir.c:713:ll_dir_get_default_layout()) unknown magic: CD40CD0
Lustre: 18052:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 32991, rollback = 7
Lustre: 18052:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 12119 previous similar messages
Lustre: 18052:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 767/3068/0, destroy: 1/4/0
Lustre: 18052:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 12184 previous similar messages
Lustre: 18052:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 767/0/0
Lustre: 18052:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 12184 previous similar messages
Lustre: 18052:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 3836/32991/0, punch: 0/0/0, quota 10/74/0
Lustre: 18052:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 12184 previous similar messages
Lustre: 18052:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 767/13039/0, delete: 2/5/0
Lustre: 18052:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 12184 previous similar messages
Lustre: 18052:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0
Lustre: 18052:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 12184 previous similar messages
Lustre: 17384:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 610, rollback = 2
Lustre: 17384:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 63 previous similar messages
Lustre: 4145:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880087efcb40 x1710260894533632/t0(0) o10->lustre-OST0003-osc-ffff880299ab4138@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631032228 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 17441:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x80:0x0] with magic=0xbd60bd0
Lustre: 17841:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x53:0x0] with magic=0xbd60bd0
Lustre: 17841:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: ll_ost00_000: service thread pid 12702 was inactive for 62.171 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 12721, comm: ll_ost05_002 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 12720, comm: ll_ost05_001 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 2 previous similar messages
Pid: 12702, comm: ll_ost00_000 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost05_004: service thread pid 19779 was inactive for 63.006 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: ll_ost05_008: service thread pid 22124 was inactive for 62.047 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 3 previous similar messages
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880273770040/0xc6b178ed82533323 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x7c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xc6b178ed8253329e expref: 986 pid: 12707 timeout: 421 lvb_type: 0
LustreError: 11-0: lustre-OST0002-osc-ffff880299ab4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff880299ab4138: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff880299ab4138: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 21747:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880089c08040 x1710260895513280/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
Lustre: ll_ost01_000: service thread pid 12705 was inactive for 62.114 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
LustreError: 19169:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88008af7f700 x1710260895527936/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 19169:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8800852fd2c0/0xc6b178ed8252cb34 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x6:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->524287) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0xc6b178ed8252cb26 expref: 816 pid: 12707 timeout: 423 lvb_type: 0
LustreError: 23344:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007b9abec0 x1710260895552384/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 23344:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: 13874:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032199 with bad export cookie 14317357651876395617
LustreError: 11-0: lustre-OST0003-osc-ffff880299ab4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff880299ab4138: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880299ab4138: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000404:0xf8:0x0]// may get corrupted (rc -108)
Lustre: 4138:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0xf9:0x0]// may get corrupted (rc -108)
Lustre: 4139:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000404:0x107:0x0]/ may get corrupted (rc -108)
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000404:0xd7:0x0]// may get corrupted (rc -108)
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000403:0x11d:0x0]/ may get corrupted (rc -108)
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000404:0xe4:0x0]/ may get corrupted (rc -108)
Lustre: 4139:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000404:0xff:0x0]// may get corrupted (rc -108)
LustreError: 25108:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880299ab4138: namespace resource [0x380000400:0x6:0x0].0x0 (ffff88027c914f40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff880299ab4138: Connection restored to 192.168.123.205@tcp (at 0@lo)
Lustre: ll_ost01_006: service thread pid 19662 was inactive for 66.033 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
sched: RT throttling activated
Lustre: 17973:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 510 < left 797, rollback = 7
Lustre: 17973:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 22378 previous similar messages
Lustre: 11128:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1293/5172/0, destroy: 1/4/1
Lustre: 11128:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 22611 previous similar messages
Lustre: 11128:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 1301/283/1
Lustre: 11128:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 22611 previous similar messages
Lustre: 11128:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 6466/55609/42, punch: 0/0/0, quota 6/6/0
Lustre: 11128:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 22611 previous similar messages
Lustre: 11128:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1295/22013/1, delete: 3/6/1
Lustre: 11128:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 22611 previous similar messages
Lustre: 17973:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0
Lustre: 17973:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 22708 previous similar messages
Lustre: 13165:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 6850, rollback = 9
Lustre: 13165:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 35 previous similar messages
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff880268385a40/0xc6b178ed82596505 lrc: 3/0,0 mode: PW/PW res: [0xdd:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 786432->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xc6b178ed825964d4 expref: 2273 pid: 20106 timeout: 522 lvb_type: 0
LustreError: 11002:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800850be440 x1710260899258432/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11002:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: 11-0: lustre-OST0001-osc-ffff8802998a8958: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff8802998a8958: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff8802998a8958: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 17526:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032300 with bad export cookie 14317357651876396842
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 17894:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8800ab885428 x1710260899187328/t0(0) o35->6c066995-9548-470f-a292-69335ad47963@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'ls.0'
LustreError: 167-0: lustre-OST0003-osc-ffff8802998a8958: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 23344:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802728a4b40 x1710260899374656/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 23344:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 12 previous similar messages
Lustre: 4152:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000403:0xf0:0x0]// may get corrupted (rc -108)
Lustre: 4151:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000404:0x1a3:0x0]// may get corrupted (rc -108)
5[27176]: segfault at 0 ip (null) sp 00007ffe1eec3c98 error 14 in 5[400000+6000]
Lustre: 4147:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000403:0xcd:0x0]// may get corrupted (rc -108)
LustreError: 27615:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802998a8958: namespace resource [0x154:0x0:0x0].0x0 (ffff880084bd1bc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection restored to 192.168.123.205@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 27561:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff8802998a8958: namespace resource [0x1e0:0x0:0x0].0x0 (ffff88027848ed40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 27561:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 7 previous similar messages
Lustre: lustre-OST0001-osc-ffff8802998a8958: Connection restored to 192.168.123.205@tcp (at 0@lo)
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880277a75680/0xc6b178ed825a7461 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000480020020 nid: 0@lo remote: 0xc6b178ed825a7430 expref: 1501 pid: 14399 timeout: 543 lvb_type: 0
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 11-0: lustre-OST0002-osc-ffff880299ab4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff880299ab4138: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff880299ab4138: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 4147:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000404:0x1b3:0x0]// may get corrupted (rc -108)
Lustre: 4147:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000403:0xfa:0x0]/ may get corrupted (rc -108)
Lustre: 4152:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88028c3c1940 x1710260900241472/t0(0) o10->lustre-OST0000-osc-ffff8802998a8958@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631032388 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
LustreError: 28118:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff880299ab4138: namespace resource [0x340000401:0xcc6:0x0].0x0 (ffff88007b823240) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 28118:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 223 previous similar messages
Lustre: lustre-OST0002-osc-ffff880299ab4138: Connection restored to 192.168.123.205@tcp (at 0@lo)
LustreError: 12352:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 12352:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11495:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 11495:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 17522:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 508 < left 14630, rollback = 7
Lustre: 17522:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 64357 previous similar messages
Lustre: 17522:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 340/1360/0, destroy: 1/4/1
Lustre: 17522:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 64217 previous similar messages
Lustre: 17522:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 344/148/0
Lustre: 17522:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 64217 previous similar messages
Lustre: 17522:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1701/14630/0, punch: 0/0/0, quota 10/90/0
Lustre: 17522:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 64217 previous similar messages
Lustre: 17522:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 341/5796/0, delete: 2/5/1
Lustre: 17522:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 64217 previous similar messages
Lustre: 17522:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 17522:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 64120 previous similar messages
LustreError: 11495:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 11495:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 11324:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 550, rollback = 2
Lustre: 11324:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 97 previous similar messages
LustreError: 30862:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880299ab4138: inode [0x240000404:0x1c1:0x0] mdc close failed: rc = -13
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 112s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88027fad6580/0xc6b178ed82606e2e lrc: 3/0,0 mode: PW/PW res: [0x152:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xc6b178ed82606e12 expref: 1335 pid: 25329 timeout: 670 lvb_type: 0
LustreError: 20690:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff8800abf2ae98 ns: filter-lustre-OST0003_UUID lock: ffff88007da04000/0xc6b178ed8265a831 lrc: 2/0,0 mode: --/PW res: [0x179:0x0:0x0].0x0 rrc: 7 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0xc6b178ed8265a823 expref: -99 pid: 20690 timeout: 0 lvb_type: 0
LustreError: 17529:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88027275a5c0 x1710260904858368/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 17529:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
LustreError: 11001:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032457 with bad export cookie 14317357651877341793
LustreError: 11-0: lustre-OST0003-osc-ffff8802998a8958: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff8802998a8958: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4151:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0x1ea:0x0]// may get corrupted (rc -108)
LustreError: 32071:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802998a8958: namespace resource [0x7e0:0x0:0x0].0x0 (ffff880264a816c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 32071:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection restored to 192.168.123.205@tcp (at 0@lo)
LustreError: 27644:0:(lustre_lmv.h:438:lmv_is_sane()) insane LMV: magic=0xcd40cd0 count=2 index=1 hash=crush:0x82000003 version=0 migrate offset=1 migrate hash=crush:3.
Lustre: 26051:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x234:0x0] with magic=0xbd60bd0
Lustre: 26051:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 32052:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '4' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 4' to finish migration.
3[1554]: segfault at 8 ip 00007f6667ee97e8 sp 00007ffea44e0a90 error 4 in ld-2.17.so[7f6667ede000+22000]
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff88007da03100/0xc6b178ed8265a8e7 lrc: 3/0,0 mode: PW/PW res: [0x17b:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xc6b178ed8265a8c4 expref: 1800 pid: 23336 timeout: 785 lvb_type: 0
LustreError: 20718:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880254f88cc0 x1710260908933504/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 20718:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 136 previous similar messages
LustreError: 11-0: lustre-OST0000-osc-ffff880299ab4138: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff880299ab4138: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 4137:0:(osc_request.c:1035:osc_init_grant()) lustre-OST0000-osc-ffff880299ab4138: granted 8437760 but already consumed 10137600
LustreError: 167-0: lustre-OST0000-osc-ffff880299ab4138: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 4141:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0x1be:0x0]// may get corrupted (rc -108)
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000403:0x1a1:0x0]/ may get corrupted (rc -108)
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0x246:0x0]/ may get corrupted (rc -108)
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0x201:0x0]// may get corrupted (rc -108)
LustreError: 2591:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880299ab4138: namespace resource [0xab0:0x0:0x0].0x0 (ffff88026f50afc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 2591:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 151 previous similar messages
Lustre: lustre-OST0000-osc-ffff880299ab4138: Connection restored to 192.168.123.205@tcp (at 0@lo)
Lustre: dir [0x240000403:0x289:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000404:0x216:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 12337:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 1458, rollback = 7
Lustre: 12337:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 76206 previous similar messages
Lustre: 12337:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0
Lustre: 12337:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 76405 previous similar messages
Lustre: 12337:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 17820:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 17820:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 76406 previous similar messages
Lustre: 17820:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 17820:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 76405 previous similar messages
Lustre: 17820:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 17820:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 76406 previous similar messages
Lustre: 12337:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 76408 previous similar messages
Lustre: 4153:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802606b1f80 x1710260911800256/t0(0) o10->lustre-OST0000-osc-ffff8802998a8958@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631032700 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
LustreError: 11153:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '17' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 17' to finish migration.
Lustre: 17482:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 10930, rollback = 9
Lustre: 17482:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 178 previous similar messages
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880272f087c0/0xc6b178ed8266eda3 lrc: 3/0,0 mode: PW/PW res: [0x146:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xc6b178ed8266ecfb expref: 1985 pid: 25329 timeout: 895 lvb_type: 0
LustreError: 26811:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032670 with bad export cookie 14317357651877800314
LustreError: 26811:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 26068:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802860f57c0 x1710260913360256/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 26068:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff8802998a8958: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 11 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff8802998a8958: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 1670:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 6c066995-9548-470f-a292-69335ad47963/ffff8802874b5d28 has 28672 pending on destroyed export
LustreError: 1670:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 228813504 != fo_tot_granted 228842176
LustreError: 1670:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 28672 != fo_tot_pending 57344
LustreError: 12750:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 228813504 != fo_tot_granted 228842176
LustreError: 815:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 12750:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 815:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 228814912 != fo_tot_granted 228843584
LustreError: 12750:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 12750:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 815:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 29082:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff880080978958 ns: filter-lustre-OST0000_UUID lock: ffff880079de4780/0xc6b178ed82732982 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x5f7:0x0].0x0 rrc: 5 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0xc6b178ed8272d7ff expref: -99 pid: 29082 timeout: 0 lvb_type: 0
LustreError: 20167:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032685 with bad export cookie 14317357651878188569
LustreError: 20167:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
Lustre: 4140:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000403:0x2d9:0x0]/ may get corrupted (rc -108)
Lustre: 4141:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x200000404:0x2bd:0x0]// may get corrupted (rc -108)
LustreError: 12741:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 242895808 != fo_tot_granted 242924480
LustreError: 12741:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 5910528 != fo_tot_pending 5939200
LustreError: 12741:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff8802998a8958: Connection restored to 192.168.123.205@tcp (at 0@lo)
LustreError: 12752:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 237013056 != fo_tot_granted 237041728
LustreError: 12752:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12752:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 12752:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
ptlrpc_watchdog_fire: 29 callbacks suppressed
Lustre: mdt03_004: service thread pid 16892 was inactive for 42.022 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16892, comm: mdt03_004 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x448/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_reint_object_lock+0x2c/0x60 [mdt]
[<0>] mdt_object_lock_save+0x29/0x50 [mdt]
[<0>] mdt_reint_rename+0x10f0/0x2ff0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 4740:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880299ab4138: namespace resource [0x815:0x0:0x0].0x0 (ffff88027d54efc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 4740:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 15 previous similar messages
Pid: 11135, comm: mdt03_001 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 12752:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 243438400 != fo_tot_granted 243467072
LustreError: 12752:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12752:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 28672 != fo_tot_pending 57344
LustreError: 12752:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 4712:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff8802998a8958: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 4712:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 12734:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 243441088 != fo_tot_granted 243469760
LustreError: 12734:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 4 previous similar messages
LustreError: 12734:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 29568 != fo_tot_pending 58240
LustreError: 12734:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 4 previous similar messages
LustreError: 5135:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff8802998a8958: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 5135:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 5135:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
Lustre: lustre-OST0000: haven't heard from client 9da291a9-717f-4193-8b26-da6e22b22b79 (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp ffff8802709d1bf8, cur 1631032734 expire 1631032704 last 1631032686
LustreError: 5499:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 5499:0:(lov_object.c:1299:lov_layout_change()) Skipped 1 previous similar message
LustreError: 5499:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x200000404:0x322:0x0] error -22.
LustreError: 5413:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 5413:0:(lov_object.c:1299:lov_layout_change()) Skipped 1 previous similar message
LustreError: 5413:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 5413:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 5413:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 5413:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 815:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 272436032 != fo_tot_granted 272464704
LustreError: 815:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 9 previous similar messages
LustreError: 815:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 200704 != fo_tot_pending 229376
LustreError: 815:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 9 previous similar messages
LustreError: 5532:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 5532:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 5532:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 4740:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880299ab4138: namespace resource [0xc30:0x0:0x0].0x0 (ffff880253186fc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff880299ab4138: Connection restored to 192.168.123.205@tcp (at 0@lo)
LustreError: 11-0: lustre-OST0000-osc-ffff880299ab4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff880299ab4138: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0000-osc-ffff880299ab4138: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 5643:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 5643:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 5643:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
Lustre: 17402:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x2b7:0x0] with magic=0xbd60bd0
Lustre: 17402:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 15984:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 15984:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 15984:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 15984:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 6209:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 6209:0:(lov_object.c:1299:lov_layout_change()) Skipped 6 previous similar messages
LustreError: 6209:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 6209:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 4 previous similar messages
Lustre: ll_ost03_001: service thread pid 12713 was inactive for 40.087 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 12713, comm: ll_ost03_001 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 6340:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 6340:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 8 previous similar messages
LustreError: 5811:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 5811:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 8 previous similar messages
LustreError: 12748:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 279790656 != fo_tot_granted 279819328
LustreError: 12748:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 25 previous similar messages
LustreError: 12748:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 458752 != fo_tot_pending 487424
LustreError: 12748:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 25 previous similar messages
LustreError: 6929:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff880299ab4138: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
LustreError: 6929:0:(lov_object.c:1299:lov_layout_change()) Skipped 13 previous similar messages
Lustre: 27653:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x312:0x0] with magic=0xbd60bd0
Lustre: 27653:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
LustreError: 6809:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 6809:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 10 previous similar messages
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff88007dbce940/0xc6b178ed826cf061 lrc: 3/0,0 mode: PW/PW res: [0x815:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->524287) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0xc6b178ed826cefdc expref: 2195 pid: 31781 timeout: 1061 lvb_type: 0
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 12313:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880285a16a80 x1710260919089344/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11001:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1631032836 with bad export cookie 14317357651876397185
LustreError: 11001:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 5 previous similar messages
LustreError: 12313:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 12 previous similar messages
LustreError: 11-0: lustre-OST0000-osc-ffff8802998a8958: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
Lustre: 4144:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000404:0x337:0x0]// may get corrupted (rc -108)
Lustre: 4145:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x280000403:0x2fc:0x0]/ may get corrupted (rc -108)
Lustre: 11001:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff880272bb9940 x1710260919108800/t0(0) o103->9da291a9-717f-4193-8b26-da6e22b22b79@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'ldlm_bl_07.0'
LustreError: 682:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 6c066995-9548-470f-a292-69335ad47963/ffff88008aeaca88 has 28672 pending on destroyed export
LustreError: 17894:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x2fe:0x0]: rc = -2
Lustre: ldlm_canceld: This server is not able to keep up with request traffic (cpu-bound).
LustreError: 10998:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 8/2s ago req@ffff880272bb9940 x1710260919108800/t0(0) o103->9da291a9-717f-4193-8b26-da6e22b22b79@0@lo:305/0 lens 328/0 e 0 to 0 dl 1631032845 ref 2 fl Interpret:H/0/ffffffff rc 0/-1 job:'ldlm_bl_07.0'
Lustre: 10998:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (8/2s); client may timeout req@ffff880272bb9940 x1710260919108800/t0(0) o103->9da291a9-717f-4193-8b26-da6e22b22b79@0@lo:305/0 lens 328/0 e 0 to 0 dl 1631032845 ref 2 fl Interpret:H/0/ffffffff rc 0/-1 job:'ldlm_bl_07.0'
Lustre: 11001:0:(service.c:1612:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=37, delay=0ms
Lustre: 11001:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-2s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880272bb9940 x1710260919108800/t0(0) o103->9da291a9-717f-4193-8b26-da6e22b22b79@0@lo:305/0 lens 328/0 e 0 to 0 dl 1631032845 ref 1 fl Interpret:H/0/ffffffff rc 0/-1 job:'ldlm_bl_07.0'
LustreError: 6819:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 6819:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 9 previous similar messages
INFO: task mv:3927 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880325722980 11088 3927 16093 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task chmod:4590 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
chmod D ffff880325721900 11344 4590 16097 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff81244207>] SyS_fchmodat+0x47/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task rm:4613 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff88008c701280 11344 4613 15955 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff8125828b>] do_unlinkat+0x13b/0x2b0
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff812592fb>] SyS_unlinkat+0x1b/0x40
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:4754 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88026e028010 11216 4754 15970 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task cat:4810 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cat D ffff880282b331c0 11360 4810 16150 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffffa1090134>] vvp_io_write_start+0xb44/0xc80 [lustre]
[<ffffffffa09d4ae9>] ? lov_lock_enqueue+0x99/0x170 [lov]
[<ffffffffa04fa627>] ? cl_lock_request+0x67/0x210 [obdclass]
[<ffffffffa04fc8ed>] cl_io_start+0x6d/0x150 [obdclass]
[<ffffffffa04fee9f>] cl_io_loop+0x9f/0x210 [obdclass]
[<ffffffffa10363cf>] ll_file_io_generic+0x33f/0xee0 [lustre]
[<ffffffffa103745c>] ll_file_aio_write+0x4ec/0x800 [lustre]
[<ffffffffa1037881>] ll_file_write+0x111/0x1e0 [lustre]
[<ffffffffa1037fc1>] ? ll_file_read+0x111/0x1e0 [lustre]
[<ffffffff81245996>] vfs_write+0xd6/0x230
[<ffffffff81245812>] ? vfs_read+0x112/0x1c0
[<ffffffff812467df>] SyS_write+0x7f/0xf0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task file_exec.sh:4822 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_exec.sh D ffff88008aca52c0 11248 4822 16378 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff8124b6b3>] vfs_fstatat+0x63/0xc0
[<ffffffff8124ba7e>] SYSC_newstat+0x2e/0x60
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
Lustre: 4152:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.205@tcp:/lustre/fid: [0x240000403:0x37d:0x0]// may get corrupted (rc -108)
[<ffffffff8124bf4e>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task rm:4953 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff880295a01280 11344 4953 16217 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff812580b5>] do_rmdir+0x165/0x200
[<ffffffff810b69ad>] ? task_work_run+0xcd/0xf0
[<ffffffff81259305>] SyS_unlinkat+0x25/0x40
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task chmod:5181 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
chmod D ffff88028c1d24f0 11344 5181 15987 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff81244207>] SyS_fchmodat+0x47/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task getfattr:5245 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
getfattr D ffff88029445a140 12736 5245 16271 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff8124b6b3>] vfs_fstatat+0x63/0xc0
[<ffffffff8124bae1>] SYSC_newlstat+0x31/0x60
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8124bf5e>] SyS_newlstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task ln:5282 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ln D ffff880293f71900 12760 5282 16051 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff8124b6b3>] vfs_fstatat+0x63/0xc0
[<ffffffff8124ba7e>] SYSC_newstat+0x2e/0x60
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8124bf4e>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 7347:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff8802998a8958: namespace resource [0x1550:0x0:0x0].0x0 (ffff8802749b34c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 7347:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 137 previous similar messages
Lustre: lustre-OST0000: haven't heard from client 6c066995-9548-470f-a292-69335ad47963 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880255ef2e98, cur 1631032884 expire 1631032854 last 1631032837
LustreError: 7707:0:(lov_object.c:1299:lov_layout_change()) lustre-clilov-ffff8802998a8958: cannot apply new layout on [0x200000404:0x322:0x0] : rc = -22
Lustre: 4153:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631032837/real 1631032837] req@ffff88025724c500 x1710260919108800/t0(0) o103->lustre-OST0001-osc-ffff880299ab4138@0@lo:17/18 lens 328/224 e 0 to 1 dl 1631032870 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'ldlm_bl_07.0'
Lustre: lustre-OST0001-osc-ffff880299ab4138: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 7707:0:(lov_object.c:1299:lov_layout_change()) Skipped 9 previous similar messages
Lustre: lustre-OST0001: Client 9da291a9-717f-4193-8b26-da6e22b22b79 (at 0@lo) reconnecting
Lustre: lustre-OST0001-osc-ffff880299ab4138: Connection restored to 192.168.123.205@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 7803:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000404:0x322:0x0]: rc = -22
LustreError: 7803:0:(lcommon_cl.c:196:cl_file_inode_init()) Skipped 9 previous similar messages
LustreError: 11149:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '7' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 7' to finish migration.
LustreError: 167-0: lustre-OST0000-osc-ffff8802998a8958: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
Lustre: 26049:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x335:0x0] with magic=0xbd60bd0
Lustre: 26049:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 7293:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -22
LustreError: 7293:0:(llite_lib.c:2983:ll_prep_inode()) Skipped 8 previous similar messages
Lustre: ll_ost05_037: service thread pid 31787 was inactive for 40.324 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 14 previous similar messages
Lustre: mdt05_004: service thread pid 17469 was inactive for 54.057 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 7 previous similar messages
LustreError: 12749:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 84143552 != fo_tot_granted 84172224
LustreError: 12749:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 71 previous similar messages
LustreError: 12749:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 12749:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 71 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 3 PID: 26746 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi crct10dif_pclmul crct10dif_common drm crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata i2c_core floppy
CPU: 3 PID: 26746 Comm: mdt01_013 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0c24262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0c0433d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0c04437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cc8e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04ee6f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa04ad6b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa049a760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa04a04ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04975ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04c8a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa083160d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa0801892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0ee4f3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0db9408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0d9ab2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04ed375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0d9bb86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0e6d0d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0e2ce53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0e2d087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0e0255c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0e0f657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07f026e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa033545e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa079b990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa079d529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffffa079c950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 647798641f8bde2f ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-0): ldiskfs_getblk:888: inode #206: block 53380: comm mdt01_013: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-0-8.
LDISKFS-fs (dm-0): Remounting filesystem read-only
LustreError: 26746:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0000: error reading offset 204800 (block 50, size 40, offs 204784), credits 0/0: rc = -28
LustreError: 26746:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0000-osd: write updates failed: rc = -28
LustreError: 26746:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 26746:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0000: can't update reply_data file: rc = -30
LustreError: 26746:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0000: failed in transaction hook: rc = -30
LustreError: 13739:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0001-osc-MDT0000: fail to cancel 23 llog-records: rc = -30
LustreError: 8688:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LDISKFS-fs error (device dm-0) in osd_trans_stop:2082: error 28
LustreError: 13739:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel 23 records: rc = -30
LustreError: 13739:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0001-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 13739:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0001-osc-MDT0000: can't cancel record: rc = -30
LustreError: 26746:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0000: failed to stop transaction: rc = -28
LustreError: 10941:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff8802796414d8 commit error: 2
LDISKFS-fs error (device dm-0) in osd_trans_stop:2082: IO failure
LustreError: 26746:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0000-osd: stop trans failed: rc = -30
LustreError: 8754:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8754:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 3 previous similar messages
Lustre: ll_ost06_030: service thread pid 5603 was inactive for 86.155 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 8309:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8309:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 1 previous similar message
Lustre: 4143:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880251803240 x1710260924035904/t0(0) o10->lustre-OST0000-osc-ffff8802998a8958@0@lo:6/4 lens 440/432 e 0 to 0 dl 1631033034 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: ll_ost06_024: service thread pid 5535 was inactive for 89.494 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 8914:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8914:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 2 previous similar messages
LustreError: 11004:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 116s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff8802778b6d00/0xc6b178ed827a4478 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x691:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xc6b178ed827a4471 expref: 1779 pid: 26418 timeout: 1186 lvb_type: 0
LustreError: 20913:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880257600040 x1710260924118080/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 20913:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 21 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff880299ab4138: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 14 previous similar messages
LustreError: 14411:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0003-osc-MDT0000: can't cancel record: rc = -30
LustreError: 14411:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 1 previous similar message
LustreError: 6083:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 9da291a9-717f-4193-8b26-da6e22b22b79/ffff8800786d2e98 has 28672 pending on destroyed export
LustreError: 8782:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 8782:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 7 previous similar messages
Lustre: 17488:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000404:0x372:0x0] with magic=0xbd60bd0
Lustre: 17488:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
6[8812]: segfault at 0 ip (null) sp 00007ffd422c0d08 error 14 in 6[400000+6000]
Lustre: 4142:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1631032890/real 1631032890] req@ffff880078107700 x1710260920730944/t0(0) o103->lustre-OST0001-osc-ffff880299ab4138@0@lo:17/18 lens 328/224 e 0 to 1 dl 1631032939 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'ldlm_bl_05.0'
Lustre: lustre-OST0001: Client 9da291a9-717f-4193-8b26-da6e22b22b79 (at 0@lo) reconnecting
LustreError: 24467:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x408:0x0] doesn't exist!: rc = -14
LustreError: 11119:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x408:0x0] doesn't exist!: rc = -14
LustreError: 11119:0:(mdt_open.c:1237:mdt_cross_open()) Skipped 5 previous similar messages
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 12 previous similar messages
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 11 previous similar messages
LustreError: 9372:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 9372:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 4 previous similar messages
LustreError: 19828:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x408:0x0] doesn't exist!: rc = -14
LustreError: 19828:0:(mdt_open.c:1237:mdt_cross_open()) Skipped 1 previous similar message
LustreError: 8859:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
Lustre: 4143:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880251803240 x1710260925262272/t0(0) o10->lustre-OST0000-osc-ffff8802998a8958@0@lo:6/4 lens 440/432 e 1 to 0 dl 1631033061 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 11952:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 9 previous similar messages
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 11952:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 9 previous similar messages
LustreError: 16071:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 16071:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 4 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1051e52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 800000028b3a6067 PUD 28b3a7067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm crct10dif_pclmul crct10dif_common crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy
CPU: 6 PID: 25076 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8802912324f0 ti: ffff88024ad2c000 task.ti: ffff88024ad2c000
RIP: 0010:[<ffffffffa1051e52>] [<ffffffffa1051e52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff88024ad2fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff880253c01458 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880327331138 RDI: ffff880327331118
RBP: ffff88024ad2fc48 R08: ffff8802668e2058 R09: 0000000000000001
R10: 0000000000000000 R11: ffff88024ad2f5e6 R12: 0000000000000000
R13: ffff8800848d48e8 R14: ffff88029b7166d8 R15: 0000000000000030
FS: 00007fa279316740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000266662000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa103f561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff8115260f>] ? delayacct_end+0x8f/0xb0
[<ffffffff81152744>] ? __delayacct_blkio_end+0x34/0x60
[<ffffffff817e0257>] ? io_schedule_timeout+0xe7/0x130
[<ffffffff811b62dd>] ? find_get_pages_tag+0x10d/0x260
[<ffffffff811c3691>] ? pagevec_lookup_tag+0x21/0x30
[<ffffffff811b400e>] ? __filemap_fdatawait_range+0xbe/0x1a0
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81264d2b>] ? iput+0x3b/0x180
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: lustre-OST0000-osc-ffff88029a0e8008: disconnect after 20s idle
Lustre: 11112:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 828, rollback = 7
Lustre: 11112:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11112:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11112:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 819/828/0, punch: 0/0/0, quota 3/3/0
Lustre: 11112:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11112:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11112:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 511 < left 827, rollback = 7
Lustre: 11112:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11112:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11112:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 819/828/1, punch: 0/0/0, quota 3/3/0
Lustre: 11112:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11112:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11112:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 299 previous similar messages
Lustre: 11112:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 511 < left 827, rollback = 7
Lustre: 11112:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 242 previous similar messages
Lustre: 11112:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11112:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 242 previous similar messages
Lustre: 11112:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11112:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 242 previous similar messages
Lustre: 11112:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 819/828/1, punch: 0/0/0, quota 3/3/0
Lustre: 11112:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 242 previous similar messages
Lustre: 11112:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11112:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 242 previous similar messages
Lustre: 11112:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11112:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 242 previous similar messages
Lustre: 11100:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 506 < left 894, rollback = 2
Lustre: 11100:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/2, destroy: 1/4/0
Lustre: 11100:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 401 previous similar messages
Lustre: 11100:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/894/0
Lustre: 11100:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 401 previous similar messages
Lustre: 11100:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1/10/0, punch: 0/0/0, quota 6/6/0
Lustre: 11100:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 401 previous similar messages
Lustre: 11100:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/194/4, delete: 3/6/1
Lustre: 11100:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 401 previous similar messages
Lustre: 11100:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 3/3/0
Lustre: 11100:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 401 previous similar messages
Lustre: 11100:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 894, rollback = 2
Lustre: 11100:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11086:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 507 < left 610, rollback = 2
Lustre: 11086:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 6 previous similar messages
cat (16892) used greatest stack depth: 10240 bytes left
Lustre: 17299:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 502 < left 671, rollback = 2
Lustre: 17299:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 17299:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0
Lustre: 17299:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 17299:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/671/0
Lustre: 17299:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 17299:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 17299:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 17299:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 17299:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 17299:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 17299:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 874, rollback = 7
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 400 previous similar messages
LustreError: 17991:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88009e7fb7e8: inode [0x200000404:0x19:0x0] mdc close failed: rc = -13
Lustre: 17963:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 507 < left 760, rollback = 2
Lustre: 17963:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 17341:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 494 < left 1329, rollback = 7
Lustre: 17341:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 4771 previous similar messages
Lustre: 11094:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 190/760/0, destroy: 0/0/0
Lustre: 11094:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 4933 previous similar messages
Lustre: 11094:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 192/15/0
Lustre: 11094:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 4933 previous similar messages
Lustre: 11094:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 951/8180/0, punch: 0/0/0, quota 14/174/3
Lustre: 11094:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 4933 previous similar messages
Lustre: 11094:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 190/3230/0, delete: 0/0/0
Lustre: 11094:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 4933 previous similar messages
Lustre: 11094:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11094:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 4933 previous similar messages
Lustre: 4120:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88008e8a3240 x1710205953078720/t0(0) o10->lustre-OST0000-osc-ffff88029a0e8008@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630979683 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 16762:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 503 < left 1250, rollback = 2
Lustre: 16762:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 16729:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 784, rollback = 7
Lustre: 16729:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1667 previous similar messages
Lustre: 17637:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0
Lustre: 17637:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 6551 previous similar messages
Lustre: 17637:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 17637:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 6552 previous similar messages
Lustre: 17637:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1553/1562/0, punch: 0/0/0, quota 6/6/2
Lustre: 17637:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 6552 previous similar messages
Lustre: 17637:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0
Lustre: 17637:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 6552 previous similar messages
Lustre: 17637:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17637:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 6552 previous similar messages
Lustre: 11107:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 505 < left 1562, rollback = 7
Lustre: 11107:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 12936 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 11 PID: 11132 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm crct10dif_pclmul crct10dif_common crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy
CPU: 11 PID: 11132 Comm: mdt_rdpg05_001 Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0be94ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0bda9eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0bdd417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0bfdaa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0bfdad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0c92d08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04c559d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04c9cf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0e4aaf4>] mdt_mfd_close+0x1f4/0x870 [mdt]
[<ffffffffa0e50e14>] mdt_close_internal+0xb4/0x240 [mdt]
[<ffffffffa0e51224>] mdt_close+0x284/0x850 [mdt]
[<ffffffffa07cc2d4>] ? tgt_request_preprocess.isra.27+0xf4/0x850 [ptlrpc]
[<ffffffffa07cd26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa03b145e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0778990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa077a529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0779950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace dc8986bfb67aba52 ]---
Lustre: 19174:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 500 < left 788, rollback = 2
Lustre: 19174:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 14 previous similar messages
Lustre: 19174:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/8, destroy: 0/0/0
Lustre: 19174:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 17872 previous similar messages
Lustre: 19174:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 10/788/0
Lustre: 19174:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 17872 previous similar messages
Lustre: 19174:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 19174:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 17872 previous similar messages
Lustre: 19174:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 19174:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 17872 previous similar messages
Lustre: 19174:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 19174:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 17872 previous similar messages
Lustre: 18995:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630979682/real 1630979682] req@ffff88029cd79300 x1710205953123392/t0(0) o101->lustre-MDT0000-mdc-ffff88009e7fb7e8@0@lo:12/10 lens 576/17800 e 0 to 1 dl 1630979728 ref 2 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'rm.0'
Lustre: lustre-MDT0000-mdc-ffff88009e7fb7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0000: Client 0ac4ffd4-9906-4a09-82f1-8303128482e2 (at 0@lo) reconnecting
Lustre: lustre-MDT0000-mdc-ffff88009e7fb7e8: Connection restored to 192.168.123.100@tcp (at 0@lo)
Lustre: 4116:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88029d2ef0c0 x1710205955484224/t0(0) o10->lustre-OST0003-osc-ffff88009e7fb7e8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630979793 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 17285:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 506 < left 29680, rollback = 7
Lustre: 17285:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 9941 previous similar messages
sched: RT throttling activated
Lustre: 19174:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 509 < left 3246, rollback = 9
Lustre: 19174:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 37 previous similar messages
LustreError: 10969:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88027901d2c0/0xb337f8daa20e8926 lrc: 3/0,0 mode: PW/PW res: [0x115:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb337f8daa20e88cb expref: 1128 pid: 12691 timeout: 402 lvb_type: 0
LustreError: 19579:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880273090cc0 x1710205957669120/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 12692:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88028feeea80 x1710205957692160/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 12692:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88029a0e8008: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff88029a0e8008: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff88029a0e8008: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 4108:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x240000403:0xb7:0x0]// may get corrupted (rc -108)
Lustre: 4109:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x200000404:0xb5:0x0]// may get corrupted (rc -108)
Lustre: 4109:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x240000403:0x9e:0x0]// may get corrupted (rc -108)
Lustre: 4108:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x200000403:0x25:0x0]// may get corrupted (rc -108)
Lustre: 4109:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x280000404:0x51:0x0]/ may get corrupted (rc -108)
LustreError: 10969:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88028de16580/0xb337f8daa20cfff2 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x6:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1310719) gid 0 flags: 0x60000480020020 nid: 0@lo remote: 0xb337f8daa20cffac expref: 1323 pid: 12703 timeout: 410 lvb_type: 0
LustreError: 11-0: lustre-OST0003-osc-ffff88009e7fb7e8: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff88009e7fb7e8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff88009e7fb7e8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4114:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x200000404:0xbc:0x0]/ may get corrupted (rc -5)
Lustre: 4114:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x240000404:0xe6:0x0]// may get corrupted (rc -108)
Lustre: 4107:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x280000403:0x29:0x0]// may get corrupted (rc -5)
Lustre: 4111:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x240000404:0xb6:0x0]/ may get corrupted (rc -108)
Lustre: 4111:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x280000403:0x63:0x0]/ may get corrupted (rc -108)
Lustre: 21437:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/12, destroy: 0/0/0
Lustre: 21437:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 23656 previous similar messages
Lustre: 21437:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/671/0
Lustre: 21437:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 23656 previous similar messages
Lustre: 21437:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 21437:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 23656 previous similar messages
Lustre: 21437:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 21437:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 23656 previous similar messages
Lustre: 21437:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 21437:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 23655 previous similar messages
Lustre: 4110:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.100@tcp:/lustre/fid: [0x280000403:0x76:0x0]// may get corrupted (rc -5)
LustreError: 22220:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029a0e8008: namespace resource [0x340000400:0xe0:0x0].0x0 (ffff88026f3d96c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffff88029a0e8008: Connection restored to 192.168.123.100@tcp (at 0@lo)
LustreError: 22261:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88009e7fb7e8: namespace resource [0x380000400:0xe0:0x0].0x0 (ffff880283824040) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 22261:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 13 previous similar messages
Lustre: 17341:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 510 < left 29680, rollback = 7
Lustre: 17341:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 23596 previous similar messages
LustreError: 18861:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '1' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 1' to finish migration.
Lustre: 14273:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 512 < left 21691, rollback = 9
Lustre: 14273:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 73 previous similar messages
LustreError: 10969:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff88007af21300/0xb337f8daa213e7f0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xc7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xb337f8daa213e7cd expref: 2035 pid: 19588 timeout: 520 lvb_type: 0
LustreError: 21373:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880270b3d7c0 x1710205961467008/t0(0) o106->lustre-OST0000@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 21373:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
LustreError: 11-0: lustre-OST0000-osc-ffff88009e7fb7e8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 3 previous similar messages
Lustre: lustre-OST0000-osc-ffff88009e7fb7e8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 20642:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88024a64f700 x1710205961541632/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 20642:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 40 previous similar messages
LustreError: 167-0: lustre-OST0000-osc-ffff88009e7fb7e8: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: 24579:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff88009e7fb7e8: namespace resource [0x2c0000401:0x7cf:0x0].0x0 (ffff880265ca34c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 24579:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 579 previous similar messages
Lustre: lustre-OST0000-osc-ffff88009e7fb7e8: Connection restored to 192.168.123.100@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: mdt00_006: service thread pid 17514 was inactive for 52.937 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 17514, comm: mdt00_006 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] lod_object_lock+0x351/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_reint_striped_lock+0x378/0x5b0 [mdt]
[<0>] mdt_migrate_parent_lock+0x3f/0xb0 [mdt]
[<0>] mdt_reint_migrate+0x8b7/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 16560:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1552/6208/0, destroy: 1/4/0
Lustre: 16560:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 32448 previous similar messages
Lustre: 16560:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 1556/148/0
Lustre: 16560:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 32448 previous similar messages
Lustre: 16560:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 7761/66746/0, punch: 0/0/0, quota 8/56/0
Lustre: 16560:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 32448 previous similar messages
Lustre: 16560:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1553/26400/0, delete: 2/5/1
Lustre: 16560:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 32448 previous similar messages
Lustre: 16560:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 16560:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 32448 previous similar messages
Lustre: 17637:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 508 < left 15533, rollback = 7
Lustre: 17637:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 38590 previous similar messages
Lustre: 4119:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880272ef8040 x1710205963279744/t0(0) o10->lustre-OST0000-osc-ffff88009e7fb7e8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630979991 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
Lustre: ll_ost03_007: service thread pid 18957 was inactive for 62.113 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 18957, comm: ll_ost03_007 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
------------[ cut here ]------------
WARNING: CPU: 1 PID: 16757 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm crct10dif_pclmul crct10dif_common crc32c_intel drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy
CPU: 1 PID: 16757 Comm: mdt00_005 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0c18262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0bf833d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0bf8437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cbce97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04cb6f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa048a6b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa0477760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa047d4ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04745ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04a5a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa080e60d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa07de892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0ef7f3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0dcc408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0dadb2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04ca375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0daeb86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0e800d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0e3fe53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0e40087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0e1555c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0e22657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07cd26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa03b145e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0778990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa077a529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffffa0779950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace dc8986bfb67aba53 ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-1): ldiskfs_getblk:888: inode #166: block 20753: comm mdt00_005: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-1-8.
LDISKFS-fs (dm-1): Remounting filesystem read-only
LustreError: 16757:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0001: error reading offset 36864 (block 9, size 48, offs 36856), credits 0/0: rc = -28
LustreError: 16757:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -28
LustreError: 16757:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 16757:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0001: can't update reply_data file: rc = -30
LustreError: 16757:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0001: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-1) in osd_trans_stop:2082: error 28
LustreError: 16757:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0001: failed to stop transaction: rc = -28
LustreError: 16757:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0001-osd: stop trans failed: rc = -30
LustreError: 11881:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff88026fc904a8 commit error: 2
LustreError: 11451:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 11451:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 12314:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 12314:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 12314:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 12314:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: mdt07_004: service thread pid 16512 was inactive for 66.169 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16512, comm: mdt07_004 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_reint_object_lock+0x2c/0x60 [mdt]
[<0>] mdt_reint_striped_lock+0x89/0x5b0 [mdt]
[<0>] mdt_migrate_parent_lock+0x3f/0xb0 [mdt]
[<0>] mdt_reint_migrate+0x8b7/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost01_005: service thread pid 18363 was inactive for 64.140 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
INFO: task mv:21640 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff8802931a73c0 11216 21640 16100 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:21645 blocked for more than 120 seconds.
Lustre: ll_ost01_012: service thread pid 20019 was inactive for 64.243 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 14 previous similar messages
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880276068010 11920 21645 17175 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:21812 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880287141900 12656 21812 16664 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:21998 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880274e10608 11344 21998 16458 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:22841 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88028ced8010 11216 22841 16476 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:22867 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88028af93760 11216 22867 16898 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
Lustre: ll_ost01_029: service thread pid 24939 was inactive for 64.210 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 8 previous similar messages
Lustre: ll_ost01_050: service thread pid 24995 was inactive for 64.003 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 22 previous similar messages
LustreError: 10969:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff880276bc3c40/0xb337f8daa2197e6d lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x7c1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xb337f8daa2197e3c expref: 1826 pid: 12694 timeout: 607 lvb_type: 0
LustreError: 17844:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88027883de00 x1710205963533056/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
Lustre: ll_ost01_057: service thread pid 25081 was inactive for 64.052 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 8 previous similar messages
LustreError: 24169:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
Lustre: lustre-OST0000-osc-ffff88029a0e8008: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0000-osc-ffff88029a0e8008: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: 25572:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff88029a0e8008: namespace resource [0x2c0000401:0x7cf:0x0].0x0 (ffff880081ef8540) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 25572:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 547 previous similar messages
Lustre: 24918:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (68/1s); client may timeout req@ffff880282e29300 x1710205962501312/t0(0) o2->adc3166f-56b4-492d-93bb-9aed3ef6d28b@0@lo:306/0 lens 440/224 e 1 to 0 dl 1630979996 ref 1 fl Complete:/0/0 rc -107/-107 job:'chown.0'
LustreError: 13198:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0000-osc-MDT0001: fail to cancel 1 llog-records: rc = -30
LustreError: 13198:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 13198:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0000-osc-MDT0001: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 13198:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
LustreError: 14050:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0001-osc-MDT0001: can't cancel 34 records: rc = -30
LustreError: 13198:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel record: rc = -30
LustreError: 14054:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0002-osc-MDT0001: can't cancel record: rc = -30
LustreError: 14054:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 3 previous similar messages
LustreError: 13198:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel 16 records: rc = -30
LustreError: 13198:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 6 previous similar messages
Lustre: 4119:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880272ef8040 x1710205963739584/t0(0) o10->lustre-OST0000-osc-ffff88009e7fb7e8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630980050 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
LustreError: 13198:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0000-osc-MDT0001: fail to cancel 42 llog-records: rc = -30
LustreError: 13198:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 37 previous similar messages
LustreError: 13198:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel 42 records: rc = -30
LustreError: 13198:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 8 previous similar messages
LustreError: 13198:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0000-osc-MDT0001: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 13198:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 21 previous similar messages
LustreError: 13198:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0000-osc-MDT0001: can't cancel record: rc = -30
LustreError: 13198:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 17 previous similar messages
LustreError: 14648:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0003-osc-MDT0001: can't cancel 113 records: rc = -30
LustreError: 14648:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 7 previous similar messages
LustreError: 14648:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0003-osc-MDT0001: can't cancel record: rc = -30
LustreError: 14648:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 1 previous similar message
LustreError: 14050:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0001-osc-MDT0001: fail to cancel 32 llog-records: rc = -30
LustreError: 14050:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 12 previous similar messages
LustreError: 14050:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0001-osc-MDT0001: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 14050:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 3 previous similar messages
LustreError: 25573:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88009e7fb7e8: namespace resource [0x340000401:0x7bf:0x0].0x0 (ffff8802497be840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 25573:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 23 previous similar messages
LustreError: 14648:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0003-osc-MDT0001: can't cancel 279 records: rc = -30
LustreError: 14648:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 15 previous similar messages
Lustre: lustre-OST0000-osc-ffff88029a0e8008: Connection restored to 192.168.123.100@tcp (at 0@lo)
LustreError: 14050:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0001-osc-MDT0001: can't cancel record: rc = -30
LustreError: 14050:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 5 previous similar messages
LustreError: 25548:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
Lustre: mdt07_002: service thread pid 11113 was inactive for 66.027 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 11-0: lustre-MDT0001-mdc-ffff88009e7fb7e8: operation ldlm_enqueue to node 0@lo failed: rc = -30
LustreError: 25697:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 25697:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 1 previous similar message

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa17aae52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 800000023e999067 PUD 29f04a067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 9 PID: 8531 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8802a47e1280 ti: ffff8802a0200000 task.ti: ffff8802a0200000
RIP: 0010:[<ffffffffa17aae52>] [<ffffffffa17aae52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff8802a0203bc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8802f7de1138 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88031428ac58 RDI: ffff88031428ac38
RBP: ffff8802a0203c48 R08: ffff8802683cf4f8 R09: ffff8802a0203a60
R10: 0000000000000006 R11: ffff8802d1db0d48 R12: 0000000000000000
R13: ffff880083633ef8 R14: ffff880274ec0118 R15: 0000000000000030
FS: 00007fa315368740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000245b96000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa1798561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810d5248>] ? __enqueue_entity+0x78/0x80
[<ffffffff810dbb8b>] ? put_prev_entity+0x8b/0x400
[<ffffffff810d5979>] ? pick_next_entity+0xa9/0x190
[<ffffffff810d7e1c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810df6c2>] ? pick_next_task_fair+0x702/0x8a0
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffff810c832d>] ? finish_task_switch+0x5d/0x1b0
[<ffffffff817e05ea>] ? __schedule+0x32a/0x7d0
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81406d25>] ? lockref_put_or_lock+0x25/0x30
[<ffffffff8126a634>] ? mntput+0x24/0x40
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: 16480:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 929, rollback = 7
Lustre: 16480:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16480:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 16480:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 920/929/0, punch: 0/0/0, quota 3/3/0
Lustre: 16480:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16480:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16480:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 511 < left 928, rollback = 7
Lustre: 16480:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16480:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 16480:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 920/929/1, punch: 0/0/0, quota 3/3/0
Lustre: 16480:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16480:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16480:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 271 previous similar messages
Lustre: 16480:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 511 < left 928, rollback = 7
Lustre: 16480:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 402 previous similar messages
Lustre: 16480:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16480:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 402 previous similar messages
Lustre: 16480:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 16480:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 402 previous similar messages
Lustre: 16480:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 920/929/1, punch: 0/0/0, quota 3/3/0
Lustre: 16480:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 530 previous similar messages
Lustre: 16480:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16480:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 530 previous similar messages
Lustre: 16480:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16480:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 530 previous similar messages
Lustre: 14983:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 728, rollback = 7
Lustre: 14983:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 244 previous similar messages
Lustre: 14983:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14983:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 244 previous similar messages
Lustre: 14983:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 14983:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 244 previous similar messages
Lustre: 14983:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 719/728/0, punch: 0/0/0, quota 3/3/0
Lustre: 14983:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 116 previous similar messages
Lustre: 14983:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14983:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 116 previous similar messages
Lustre: 14983:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14983:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 116 previous similar messages
Lustre: 18803:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 510 < left 760, rollback = 2
Lustre: 18803:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/0, destroy: 0/0/0
Lustre: 18803:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 718 previous similar messages
Lustre: 18803:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 8/760/0
Lustre: 18803:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 718 previous similar messages
Lustre: 18803:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 18803:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 718 previous similar messages
Lustre: 18803:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/2, delete: 0/0/0
Lustre: 18803:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 718 previous similar messages
Lustre: 18803:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 18803:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 718 previous similar messages
Lustre: 18803:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 509 < left 760, rollback = 2
Lustre: 19946:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 520, rollback = 7
Lustre: 19946:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 718 previous similar messages
Lustre: 19946:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 19946:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 19946:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 19946:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 19946:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 511/520/0, punch: 0/0/0, quota 3/3/0
Lustre: 19946:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 19946:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 19946:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 19946:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 19946:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 17449:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 788, rollback = 2
Lustre: 17449:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 19982:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 509 < left 3246, rollback = 9
Lustre: 19982:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 17539:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 512 < left 707, rollback = 7
Lustre: 17539:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 512 previous similar messages
Lustre: 17471:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 503 < left 671, rollback = 2
Lustre: 17471:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14974:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 920/3680/0, destroy: 1/4/0
Lustre: 14974:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 3596 previous similar messages
Lustre: 14974:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 924/148/0
Lustre: 14974:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 3596 previous similar messages
Lustre: 14974:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 4601/39570/0, punch: 0/0/0, quota 6/6/0
Lustre: 14974:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 3596 previous similar messages
Lustre: 14974:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 921/15656/0, delete: 2/5/0
Lustre: 14974:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 3596 previous similar messages
Lustre: 14974:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0
Lustre: 14974:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 3596 previous similar messages
Lustre: 19972:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 8703, rollback = 9
Lustre: 19972:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 20902:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 507 < left 26369, rollback = 7
Lustre: 20902:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3066 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 14 PID: 18541 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 14 PID: 18541 Comm: mdt07_006 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0d764ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0d679eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0d6a417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0d8aaa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0d8aad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0ca6d08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04d759d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04dbcf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0ed8e09>] mdt_getattr_name_lock+0x859/0x2590 [mdt]
[<ffffffffa0778097>] ? lustre_msg_buf+0x17/0x60 [ptlrpc]
[<ffffffffa07a2bbf>] ? __req_capsule_get+0x41f/0x6a0 [ptlrpc]
[<ffffffffa07784cc>] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc]
[<ffffffffa0ee1785>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<ffffffffa0ed528d>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<ffffffffa0ee14b0>] ? mdt_intent_layout+0xe90/0xe90 [mdt]
[<ffffffffa0ede2a1>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<ffffffffa0729962>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<ffffffff817e340e>] ? _raw_read_unlock+0xe/0x20
[<ffffffffa0300d85>] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs]
[<ffffffffa0303fb6>] ? cfs_hash_add+0xa6/0x180 [libcfs]
[<ffffffffa07537a6>] ldlm_handle_enqueue0+0xaa6/0x16d0 [ptlrpc]
[<ffffffffa0778000>] ? lustre_msg_buf_v2+0x160/0x1e0 [ptlrpc]
[<ffffffffa07d5494>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<ffffffffa07df26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa034745e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa078a990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa078c529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa078b950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 5b2c3ed042866441 ]---
Lustre: 20912:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 505 < left 805, rollback = 2
Lustre: 20912:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 11778:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880091163240 x1710150313961984/t0(0) o10->lustre-OST0002-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630926661 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 18696:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 190/760/0, destroy: 1/4/1
Lustre: 18696:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 2491 previous similar messages
Lustre: 18696:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 190/0/0
Lustre: 18696:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 2491 previous similar messages
Lustre: 18696:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 951/8180/0, punch: 0/0/0, quota 6/6/0
Lustre: 18696:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 2491 previous similar messages
Lustre: 18696:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 190/3230/0, delete: 2/5/1
Lustre: 18696:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 2491 previous similar messages
Lustre: 18696:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0
Lustre: 18696:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 2491 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff880091fbbc40/0xd30819a9e29976ff lrc: 3/0,0 mode: PW/PW res: [0x2:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->786431) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0xd30819a9e29976e3 expref: 1711 pid: 15683 timeout: 19343 lvb_type: 0
LustreError: 11-0: lustre-OST0000-osc-ffff880298c7e678: operation ost_write to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff880298c7e678: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0000-osc-ffff880298c7e678: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 11777:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000403:0x35:0x0]// may get corrupted (rc -108)
Lustre: 11776:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x200000404:0xdb:0x0]// may get corrupted (rc -108)
Lustre: 11776:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0xdc:0x0]/ may get corrupted (rc -108)
Lustre: 11776:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0xbf:0x0]/ may get corrupted (rc -108)
Lustre: 11776:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0xae:0x0]// may get corrupted (rc -108)
LustreError: 24001:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880298c7e678: namespace resource [0x10:0x0:0x0].0x0 (ffff880065f6de40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: ll_ost01_005: service thread pid 19832 was inactive for 62.115 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 19832, comm: ll_ost01_005 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 20012, comm: ll_ost01_008 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 19987, comm: ll_ost01_006 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost01_003: service thread pid 16371 was inactive for 62.258 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: ll_ost01_012: service thread pid 20864 was inactive for 62.040 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 5 previous similar messages
Lustre: ll_ost01_024: service thread pid 20935 was inactive for 62.009 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 10 previous similar messages
Lustre: ll_ost01_039: service thread pid 21091 was inactive for 62.239 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 16 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff88024fba5e00/0xd30819a9e29a5998 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->3145727) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xd30819a9e29a598a expref: 2139 pid: 19832 timeout: 19355 lvb_type: 0
Lustre: ll_ost01_055: service thread pid 21396 was inactive for 62.199 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 13 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88026d2b1a80/0xd30819a9e29c953e lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xd30819a9e29c9530 expref: 2429 pid: 19987 timeout: 19364 lvb_type: 0
Lustre: lustre-OST0000-osc-ffff8802cfdd12a8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0002-osc-ffff8802cfdd12a8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 24028:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff8802cfdd12a8: namespace resource [0x2c0000400:0x130:0x0].0x0 (ffff88021ee516c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffff8802cfdd12a8: Connection restored to 192.168.123.120@tcp (at 0@lo)
LustreError: 24028:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 142 previous similar messages
Lustre: 11778:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880091163240 x1710150314540224/t0(0) o10->lustre-OST0002-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630926755 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: mdt02_001: service thread pid 14980 was inactive for 62.134 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 9 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff8802ea317840/0xd30819a9e29d19ea lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xc7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd30819a9e29d19dc expref: 1798 pid: 15669 timeout: 19386 lvb_type: 0
LustreError: 19784:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88027d399300 x1710150314547776/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 21428:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 93/6s ago req@ffff8802a37f9f80 x1710150312596992/t0(0) o1->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:616/0 lens 440/0 e 2 to 0 dl 1630926701 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 21428:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (93/6s); client may timeout req@ffff8802a37f9f80 x1710150312596992/t0(0) o1->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:616/0 lens 440/0 e 2 to 0 dl 1630926701 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 19784:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0001-osc-ffff880298c7e678: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298c7e678: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0001-osc-ffff880298c7e678: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 20935:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 94/5s ago req@ffff8800667fd7c0 x1710150312616448/t0(0) o1->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:618/0 lens 440/0 e 2 to 0 dl 1630926703 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 19997:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (94/5s); client may timeout req@ffff8802aecc70c0 x1710150312616576/t0(0) o1->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:618/0 lens 440/0 e 2 to 0 dl 1630926703 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 19997:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 98 previous similar messages
LustreError: 20935:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 92 previous similar messages
LustreError: 24028:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff8802cfdd12a8: namespace resource [0x2c0000400:0x130:0x0].0x0 (ffff88021ee516c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 24028:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 12 previous similar messages
Lustre: lustre-OST0000-osc-ffff8802cfdd12a8: Connection restored to 192.168.123.120@tcp (at 0@lo)
LustreError: 20906:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 70/23s ago req@ffff8802ebec5e00 x1710150312989184/t0(0) o101->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:601/0 lens 328/0 e 1 to 0 dl 1630926686 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'ln.0'
LustreError: 20906:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 386 previous similar messages
Lustre: 20906:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (70/23s); client may timeout req@ffff8802ebec5e00 x1710150312989184/t0(0) o101->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:601/0 lens 328/0 e 1 to 0 dl 1630926686 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 20906:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 403 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff880298c7e678: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 14954:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630926709 with bad export cookie 15206432359258286029
Lustre: lustre-OST0002-osc-ffff880298c7e678: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 14954:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 6 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff880298c7e678: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff880298c7e678: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: 20732:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 658, rollback = 7
Lustre: 20732:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3852 previous similar messages
Lustre: 20732:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20732:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1404 previous similar messages
Lustre: 20732:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 20732:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1404 previous similar messages
Lustre: 20732:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 649/658/0, punch: 0/0/0, quota 3/3/0
Lustre: 20732:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1404 previous similar messages
Lustre: 20732:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20732:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1404 previous similar messages
Lustre: 20732:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20732:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1404 previous similar messages
Lustre: lustre-OST0001-osc-ffff880298c7e678: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: 14987:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 688, rollback = 2
Lustre: 14987:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: dir [0x280000404:0x39:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000404:0xaa:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 4 previous similar messages
Lustre: dir [0x200000404:0xc4:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 20004:0:(mdt_xattr.c:429:mdt_dir_layout_update()) lustre-MDT0002: [0x280000403:0xb9:0x0] migrate mdt index mismatch 2 != 0
LustreError: 22084:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x39:0x0]: rc = -2
Lustre: 22084:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 511 < left 27917, rollback = 7
Lustre: 22084:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 21454 previous similar messages
LustreError: 14999:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x39:0x0]: rc = -2
Lustre: 11788:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802f372c500 x1710150318122368/t0(0) o10->lustre-OST0000-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630926853 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'truncate.0'
Lustre: 17906:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 509 < left 3909, rollback = 9
Lustre: 17906:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 83 previous similar messages
Lustre: 17906:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 229/916/0, destroy: 1/4/0
Lustre: 17906:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 35233 previous similar messages
Lustre: 17906:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 233/148/0
Lustre: 17906:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 35235 previous similar messages
Lustre: 17906:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1146/9857/0, punch: 0/0/0, quota 10/74/0
Lustre: 17906:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 35237 previous similar messages
Lustre: 17906:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 230/3909/0, delete: 2/5/1
Lustre: 17906:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 35237 previous similar messages
Lustre: 17906:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 17906:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 35237 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8802d9ae6580/0xd30819a9e2a42248 lrc: 3/0,0 mode: PW/PW res: [0x1ee:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xd30819a9e2a42202 expref: 876 pid: 24507 timeout: 19530 lvb_type: 0
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff8802cfdd12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 5 previous similar messages
Lustre: lustre-OST0003-osc-ffff8802cfdd12a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 19296:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880265194500 x1710150320969280/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 19296:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 31 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff8802cfdd12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: lustre-OST0003-osc-ffff8802cfdd12a8: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: Skipped 2 previous similar messages
Lustre: 11779:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802a467b240 x1710150321808064/t0(0) o10->lustre-OST0001-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630926934 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
LustreError: 14994:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '5' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 5' to finish migration.
Lustre: 17675:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1439, rollback = 7
Lustre: 17675:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 26957 previous similar messages
Lustre: 14933:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff8802f73f7018 x1710150322771456/t0(0) o104->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:0/0 lens 328/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
LustreError: 20732:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '11' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 11' to finish migration.
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff880086d04b40/0xd30819a9e2acb1f7 lrc: 3/0,0 mode: PW/PW res: [0x1c0:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xd30819a9e2acb1cd expref: 3243 pid: 25291 timeout: 19640 lvb_type: 0
LustreError: 19296:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802c2630040 x1710150325686720/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0001-osc-ffff880298c7e678: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 4 previous similar messages
Lustre: lustre-OST0001-osc-ffff880298c7e678: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880298c7e678: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 18794:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802e8b5e440 x1710150325803712/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 24478:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880284500008 ns: filter-lustre-OST0002_UUID lock: ffff880272e7f480/0xd30819a9e2b4ac53 lrc: 5/0,0 mode: PW/PW res: [0x340000400:0x5b7:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0xd30819a9e2b4ac45 expref: 4196 pid: 24478 timeout: 0 lvb_type: 0
LustreError: 18794:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Lustre: 11785:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000404:0x16a:0x0]// may get corrupted (rc -108)
LustreError: 26154:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f6f32c00 x1710150325960064/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 26154:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 77 previous similar messages
LustreError: 30141:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298c7e678: namespace resource [0x300000400:0x3f0:0x0].0x0 (ffff8802e30b25c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 30141:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 89 previous similar messages
Lustre: lustre-OST0001-osc-ffff880298c7e678: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: 15011:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880243655180 x1710150325992960/t0(0) o35->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'chown.0'
LustreError: 30267:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff880298c7e678: namespace resource [0x340000400:0x58b:0x0].0x0 (ffff88024cf84f40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 30267:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 311 previous similar messages
Lustre: 18016:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 496 < left 849, rollback = 2
Lustre: 18016:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 38 previous similar messages
Lustre: dir [0x240000403:0x1bd:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880246c09300/0xd30819a9e2b46191 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x2b8:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xd30819a9e2b46175 expref: 1765 pid: 16483 timeout: 19746 lvb_type: 0
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 21015:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802409b3240 x1710150330318848/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 21015:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff8802cfdd12a8: operation ost_write to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff8802cfdd12a8: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0002-osc-ffff8802cfdd12a8: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
Lustre: 11782:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000403:0x204:0x0]// may get corrupted (rc -5)
LustreError: 17677:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630927069 with bad export cookie 15206432359259590857
Lustre: 11781:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000403:0x252:0x0]// may get corrupted (rc -108)
Lustre: 11780:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000404:0x149:0x0]// may get corrupted (rc -108)
Lustre: 11783:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000403:0x261:0x0]/ may get corrupted (rc -5)
LustreError: 587:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802cfdd12a8: namespace resource [0x160:0x0:0x0].0x0 (ffff8802adf62840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 587:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 69 previous similar messages
Lustre: 11782:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x200000403:0x194:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff8802cfdd12a8: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
14[572]: segfault at 8 ip 00007f64ea2007e8 sp 00007fff16aeb7f0 error 4 in ld-2.17.so[7f64ea1f5000+22000]
Lustre: 22117:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 372/1488/0, destroy: 1/4/1
Lustre: 22117:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 110424 previous similar messages
Lustre: 22117:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 376/148/0
Lustre: 22117:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 110427 previous similar messages
Lustre: 22117:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1861/16006/0, punch: 0/0/0, quota 8/56/0
Lustre: 22117:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 110427 previous similar messages
Lustre: 22117:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 373/6340/0, delete: 2/5/0
Lustre: 22117:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 110423 previous similar messages
Lustre: 22117:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 22117:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 110424 previous similar messages
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88026a9f0400/0xd30819a9e2bce71b lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x3e7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 786432->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xd30819a9e2bce6e3 expref: 1914 pid: 20064 timeout: 19849 lvb_type: 0
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 16386:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630927170 with bad export cookie 15206432359258849249
Lustre: lustre-OST0003-osc-ffff880298c7e678: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 15675:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802219ebec0 x1710150334656960/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 15675:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 47 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff880298c7e678: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
LustreError: 16386:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 3 previous similar messages
LustreError: 22314:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0003: cli 4daad2b5-7c77-41c5-b493-e6876f5214e1/ffff8802ea355d28 has 122880 pending on destroyed export
LustreError: 22314:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 40779328 != fo_tot_granted 40902208
LustreError: 22314:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 0 != fo_tot_pending 122880
LustreError: 2844:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298c7e678: namespace resource [0x160:0x0:0x0].0x0 (ffff88025f232840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 2844:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 97 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298c7e678: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 22084:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 509 < left 36904, rollback = 7
Lustre: 22084:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 109697 previous similar messages
LustreError: 11-0: lustre-MDT0002-mdc-ffff8802cfdd12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
LustreError: 31347:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cfdd12a8: inode [0x280000403:0x240:0x0] mdc close failed: rc = -108
LustreError: 31885:0:(llite_lib.c:2983:ll_prep_inode()) new_inode -fatal: rc -5
LustreError: 3140:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff8802cfdd12a8: inode [0x280000404:0x1f7:0x0] mdc close failed: rc = -108
LustreError: 3140:0:(file.c:234:ll_close_inode_openhandle()) Skipped 20 previous similar messages
LustreError: 15699:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 80710336 != fo_tot_granted 80833216
LustreError: 15699:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 90112 != fo_tot_pending 212992
LustreError: 15700:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 110729920 != fo_tot_granted 110852800
LustreError: 15700:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 15007:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000404:0x1b0:0x0]: rc = -2
LustreError: 15699:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 110729920 != fo_tot_granted 110852800
LustreError: 15699:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 15688:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 110731072 != fo_tot_granted 110853952
LustreError: 15688:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 15688:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 15688:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
6[3064]: segfault at 8 ip 00007f385bac97e8 sp 00007ffde0c3b320 error 4 in ld-2.17.so[7f385babe000+22000]
LustreError: 561:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 119074240 != fo_tot_granted 119197120
LustreError: 561:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 3 previous similar messages
LustreError: 561:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 561:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 3 previous similar messages
LustreError: 14987:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '6' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 6' to finish migration.
LustreError: 18761:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xf0:0x0]: rc = -2
LustreError: 15702:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 143175616 != fo_tot_granted 143298496
LustreError: 15702:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 5 previous similar messages
LustreError: 15702:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 15702:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 5 previous similar messages
Lustre: 26590:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 11202, rollback = 9
Lustre: 26590:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 248 previous similar messages
Lustre: 11775:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88026c114500 x1710150338737856/t0(0) o10->lustre-OST0000-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630927304 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 5431:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88024d23f0c0 x1710150338310464/t0(0) o2->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:434/0 lens 440/0 e 0 to 0 dl 1630927274 ref 2 fl New:/0/ffffffff rc 0/-1 job:'touch.0'
LustreError: 15693:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 243346880 != fo_tot_granted 243469760
LustreError: 15693:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 12 previous similar messages
LustreError: 15693:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 4277120 != fo_tot_pending 4400000
LustreError: 15693:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 12 previous similar messages
Lustre: 5197:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff88008c159940 x1710150338502912/t0(0) o101->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:436/0 lens 328/0 e 0 to 0 dl 1630927276 ref 2 fl New:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 5197:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 61 previous similar messages
Lustre: 11784:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927260/real 1630927260] req@ffff8802651b1300 x1710150338502912/t0(0) o101->lustre-OST0003-osc-ffff880298c7e678@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630927284 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: lustre-OST0003-osc-ffff880298c7e678: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0003: Client 4daad2b5-7c77-41c5-b493-e6876f5214e1 (at 0@lo) reconnecting
Lustre: lustre-OST0003-osc-ffff880298c7e678: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: dir [0x240000403:0x34d:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88006b1a1a80/0xd30819a9e2c4e870 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x53d:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->786431) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xd30819a9e2c4e5a6 expref: 549 pid: 21022 timeout: 19974 lvb_type: 0
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 11-0: lustre-OST0003-osc-ffff880298c7e678: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: 167-0: lustre-OST0003-osc-ffff880298c7e678: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 1 previous similar message
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927257/real 1630927257] req@ffff880091928680 x1710150338423744/t0(0) o2->lustre-OST0003-osc-ffff8802cfdd12a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1630927294 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'touch.0'
Lustre: lustre-OST0001: Client 717fdd9d-1a9e-495a-9c80-e8efeeb57655 (at 0@lo) reconnecting
Lustre: 11786:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000404:0x1f2:0x0]// may get corrupted (rc -108)
Lustre: 11781:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0x304:0x0]/ may get corrupted (rc -108)
Lustre: 11781:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0x316:0x0]// may get corrupted (rc -108)
Lustre: 11780:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000404:0x24c:0x0]// may get corrupted (rc -108)
Lustre: 5197:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (44/35), not sending early reply req@ffff8802aee4de00 x1710150338422336/t0(0) o2->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:462/0 lens 440/0 e 1 to 0 dl 1630927344 ref 2 fl New:/0/ffffffff rc 0/-1 job:'touch.0'
LustreError: 6685:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298c7e678: namespace resource [0x380000400:0x540:0x0].0x0 (ffff8802d5f33c40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 6685:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 13 previous similar messages
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927258/real 1630927258] req@ffff880265128cc0 x1710150338429760/t0(0) o2->lustre-OST0000-osc-ffff8802cfdd12a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1630927301 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'touch.0'
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 113 previous similar messages
Lustre: lustre-OST0000: Client 717fdd9d-1a9e-495a-9c80-e8efeeb57655 (at 0@lo) reconnecting
Lustre: Skipped 2 previous similar messages
LustreError: 15267:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 15267:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 18312:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 3' to finish migration.
Lustre: 11775:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88026c114500 x1710150340333120/t0(0) o10->lustre-OST0000-osc-ffff880298c7e678@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630927347 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927299/real 1630927299] req@ffff8800a605ac00 x1710150338430144/t0(0) o2->lustre-OST0002-osc-ffff8802cfdd12a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1630927320 ref 1 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'touch.0'
Lustre: lustre-OST0002: Client 717fdd9d-1a9e-495a-9c80-e8efeeb57655 (at 0@lo) reconnecting
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927307/real 1630927307] req@ffff880079319f80 x1710150338301888/t0(0) o2->lustre-OST0000-osc-ffff8802cfdd12a8@0@lo:28/4 lens 440/432 e 0 to 1 dl 1630927326 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'touch.0'
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 59 previous similar messages
Lustre: lustre-OST0000: Client 717fdd9d-1a9e-495a-9c80-e8efeeb57655 (at 0@lo) reconnecting
Lustre: Skipped 2 previous similar messages
ptlrpc_watchdog_fire: 78 callbacks suppressed
Lustre: ll_ost05_004: service thread pid 19765 was inactive for 76.001 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 20 previous similar messages
Lustre: ll_ost05_025: service thread pid 1261 was inactive for 76.000 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 1261, comm: ll_ost05_025 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 20360, comm: ll_ost05_009 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 30625, comm: ll_ost05_023 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 15694:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 185286848 != fo_tot_granted 185409728
LustreError: 15694:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 36 previous similar messages
LustreError: 15694:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 122880
LustreError: 15694:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 36 previous similar messages
Lustre: lustre-OST0003: haven't heard from client 4daad2b5-7c77-41c5-b493-e6876f5214e1 (at 0@lo) in 50 seconds. I think it's dead, and I am evicting it. exp ffff880066768008, cur 1630927346 expire 1630927316 last 1630927296
Lustre: 30625:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (90/9s); client may timeout req@ffff880241a40cc0 x1710150337988032/t0(0) o1->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:501/0 lens 440/0 e 0 to 0 dl 1630927341 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 30625:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 149 previous similar messages
LustreError: 24414:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800742b5e00 x1710150340770048/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 24414:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 13 previous similar messages
Lustre: 5424:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (62/33s); client may timeout req@ffff88023b29d180 x1710150338295488/t0(0) o2->717fdd9d-1a9e-495a-9c80-e8efeeb57655@0@lo:434/0 lens 440/0 e 0 to 0 dl 1630927318 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'touch.0'
Lustre: 5424:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 57 previous similar messages
LustreError: 5184:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 16/43s ago req@ffff880095800040 x1710150339621696/t0(0) o101->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:468/0 lens 328/0 e 0 to 0 dl 1630927308 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'ln.0'
LustreError: 5184:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 155 previous similar messages
LustreError: 7654:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880298c7e678: namespace resource [0x130:0x0:0x0].0x0 (ffff8802f41fa0c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 7654:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 185 previous similar messages
Lustre: dir [0x240000403:0x399:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630927321/real 1630927321] req@ffff880295a857c0 x1710150338312832/t0(0) o2->lustre-OST0002-osc-ffff8802cfdd12a8@0@lo:28/4 lens 440/432 e 2 to 1 dl 1630927370 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'touch.0'
Lustre: 11785:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Lustre: lustre-OST0002: Client 717fdd9d-1a9e-495a-9c80-e8efeeb57655 (at 0@lo) reconnecting
LustreError: 11-0: lustre-OST0000-osc-ffff8802cfdd12a8: operation ost_getattr to node 0@lo failed: rc = -107
LustreError: Skipped 42 previous similar messages
Lustre: 15671:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (93/8s); client may timeout req@ffff880099d31300 x1710150339183680/t0(0) o1->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:538/0 lens 440/432 e 2 to 0 dl 1630927378 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 15671:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 392 previous similar messages
LustreError: 30665:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802c3a32548 ns: filter-lustre-OST0001_UUID lock: ffff8802b4d8ed00/0xd30819a9e2cf6f54 lrc: 1/0,0 mode: PW/PW res: [0x300000400:0xac6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x54801000000000 nid: 0@lo remote: 0xd30819a9e2ced102 expref: 487 pid: 30665 timeout: 0 lvb_type: 0
LustreError: 29315:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 717fdd9d-1a9e-495a-9c80-e8efeeb57655/ffff88026dd612a8 has 61440 pending on destroyed export
Lustre: 30668:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (93/10s); client may timeout req@ffff88009587bec0 x1710150339183744/t0(0) o1->4daad2b5-7c77-41c5-b493-e6876f5214e1@0@lo:538/0 lens 440/432 e 2 to 0 dl 1630927378 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 30668:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 10 previous similar messages
Lustre: 11781:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x240000403:0x3b4:0x0]/ may get corrupted (rc -108)
Lustre: 11780:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000405:0x100:0x0]/ may get corrupted (rc -108)
Lustre: 11780:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x280000405:0x10c:0x0]// may get corrupted (rc -108)
Lustre: 11779:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x200000404:0x2a9:0x0]/ may get corrupted (rc -108)
Lustre: 11779:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.120@tcp:/lustre/fid: [0x200000404:0x264:0x0]/ may get corrupted (rc -108)
Lustre: 14932:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s req@ffff88025d7aa598 x1710150344480832/t0(0) o105->LOV_OSC_UUID@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:''
LustreError: 18044:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '10' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 10' to finish migration.
LustreError: 15688:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 174205888 != fo_tot_granted 174267328
LustreError: 15688:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 115 previous similar messages
LustreError: 15688:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 4220672 != fo_tot_pending 4282112
LustreError: 15688:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 115 previous similar messages
LustreError: 21064:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '10' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 10' to finish migration.
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8800913d2200/0xd30819a9e2d4fa08 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0xcc2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 1048576->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd30819a9e2d4f9f3 expref: 2038 pid: 1232 timeout: 20224 lvb_type: 0
LustreError: 14966:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 15 previous similar messages
LustreError: 14965:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88006fed1300 x1710150347545344/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 14965:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 55 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff8802cfdd12a8: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 32 previous similar messages
Lustre: lustre-OST0003-osc-ffff8802cfdd12a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 17 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff8802cfdd12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 7 previous similar messages
LustreError: 25099:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8802dabe53d8 ns: filter-lustre-OST0003_UUID lock: ffff8802f41f61c0/0xd30819a9e2d99469 lrc: 3/0,0 mode: PW/PW res: [0xd69:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000080020000 nid: 0@lo remote: 0xd30819a9e2d9945b expref: 1616 pid: 25099 timeout: 0 lvb_type: 0
LustreError: 11702:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff8802cfdd12a8: namespace resource [0x190:0x0:0x0].0x0 (ffff88009584c7c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 11702:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 655 previous similar messages
Lustre: lustre-OST0003-osc-ffff8802cfdd12a8: Connection restored to 192.168.123.120@tcp (at 0@lo)
Lustre: Skipped 17 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 8 PID: 24026 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks ata_piix crc32c_intel serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 8 PID: 24026 Comm: mdt04_010 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0da5262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0d8533d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0d85437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cd0e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04dd6f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa049c6b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa0489760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa048f4ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04865ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04b7a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa082060d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffff811beccd>] ? __free_pages+0x1d/0x30
[<ffffffff8121efe4>] ? kfree+0x1e4/0x200
[<ffffffffa07f0892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0fb3f3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0e88408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0e69b2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04dc375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0e6ab86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0f3c0d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0efbe53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0efc087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0ed155c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0ede657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07df26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa034745e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa078a990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa078c529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa078b950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 5b2c3ed042866442 ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-1): ldiskfs_getblk:888: inode #163: block 20469: comm mdt04_010: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-1-8.
LDISKFS-fs (dm-1): Remounting filesystem read-only
LustreError: 24026:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0001: error reading offset 270336 (block 66, size 40, offs 270320), credits 0/0: rc = -28
LustreError: 24026:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -28
LustreError: 24026:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 24026:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0001: can't update reply_data file: rc = -30
LustreError: 24026:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0001: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-1) in osd_trans_stop:2082: error 28
LustreError: 24026:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0001: failed to stop transaction: rc = -28
LustreError: 15250:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff88021f417788 commit error: 2
LustreError: 24026:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0001-osd: stop trans failed: rc = -30
LustreError: 24026:0:(update_trans.c:1084:top_trans_stop()) lustre-OST0003-osc-MDT0001: stop trans failed: rc = -30
LustreError: 17067:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 15072:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116
LustreError: 15072:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 12308:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1068e52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 800000029222d067 PUD 2776c1067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm crct10dif_pclmul ata_piix crct10dif_common drm_panel_orientation_quirks crc32c_intel serio_raw virtio_blk i2c_core libata floppy
CPU: 13 PID: 5065 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8800812d3760 ti: ffff88025100c000 task.ti: ffff88025100c000
RIP: 0010:[<ffffffffa1068e52>] [<ffffffffa1068e52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0000:ffff88025100fbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88007bb81db8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880320c4c258 RDI: ffff880320c4c238
RBP: ffff88025100fc48 R08: ffff88024e2f67d8 R09: ffff88025100fa60
R10: 0000000000000001 R11: ffff88028593d7c0 R12: 0000000000000000
R13: ffff8802563f4598 R14: ffff880072adcb18 R15: 0000000000000030
FS: 00007fce3bc05740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 00000002704f2000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa1056561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff811b515b>] ? unlock_page+0x2b/0x30
[<ffffffff811e5e2e>] ? do_read_fault.isra.64+0x12e/0x170
[<ffffffff811ea1cf>] ? handle_pte_fault+0x2ef/0xca0
[<ffffffff811ecf6d>] ? __handle_mm_fault+0x31d/0x720
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff81245a38>] ? vfs_write+0x178/0x230
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: lustre-OST0001-osc-ffff88029ad2a548: disconnect after 21s idle
Lustre: lustre-OST0002-osc-ffff88029ad2a548: disconnect after 22s idle
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 1172, rollback = 7
Lustre: 11111:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11111:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11111:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1163/1172/0, punch: 0/0/0, quota 3/3/0
Lustre: 11111:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11111:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 1172, rollback = 7
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 237 previous similar messages
Lustre: 11110:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11110:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 491 previous similar messages
Lustre: 11110:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11110:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 491 previous similar messages
Lustre: 11110:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 576/585/0, punch: 0/0/0, quota 3/3/0
Lustre: 11110:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 491 previous similar messages
Lustre: 11110:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11110:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 491 previous similar messages
Lustre: 11110:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11110:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 491 previous similar messages
Lustre: 11091:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 512 < left 49976, rollback = 7
Lustre: 11091:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1500 previous similar messages
Lustre: 11091:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1162/4648/0, destroy: 0/0/0
Lustre: 11091:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1246 previous similar messages
Lustre: 11091:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 1164/15/0
Lustre: 11091:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1246 previous similar messages
Lustre: 11091:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5811/49976/0, punch: 0/0/0, quota 10/106/0
Lustre: 11091:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1246 previous similar messages
Lustre: 11091:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1162/19754/0, delete: 0/0/0
Lustre: 11091:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1246 previous similar messages
Lustre: 11091:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11091:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1246 previous similar messages
Lustre: 16624:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1164, rollback = 7
Lustre: 16624:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 4661 previous similar messages
Lustre: 16624:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 16624:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 5099 previous similar messages
Lustre: 16624:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 16624:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 5099 previous similar messages
Lustre: 16624:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1155/1164/1, punch: 0/0/0, quota 3/3/0
Lustre: 16624:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 5099 previous similar messages
Lustre: 16624:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 16624:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 5099 previous similar messages
Lustre: 16624:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 16624:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 5099 previous similar messages
Lustre: 11100:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 501 < left 866, rollback = 2
Lustre: 11108:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1529, rollback = 7
Lustre: 11108:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1154 previous similar messages
Lustre: 11108:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11108:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 726 previous similar messages
Lustre: 11108:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 11108:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 726 previous similar messages
Lustre: 11108:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1520/1529/0, punch: 0/0/0, quota 3/3/0
Lustre: 11108:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 726 previous similar messages
Lustre: 11108:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11108:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 726 previous similar messages
Lustre: 11108:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11108:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 726 previous similar messages
Lustre: 17104:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 509 < left 10932, rollback = 7
Lustre: 17104:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2486 previous similar messages
Lustre: 17104:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 254/1016/0, destroy: 0/0/0
Lustre: 17104:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 2485 previous similar messages
Lustre: 17104:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 256/15/0
Lustre: 17104:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 2485 previous similar messages
Lustre: 17104:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1271/10932/0, punch: 0/0/0, quota 14/174/2
Lustre: 17104:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 2485 previous similar messages
Lustre: 17104:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 254/4318/0, delete: 0/0/0
Lustre: 17104:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 2485 previous similar messages
Lustre: 17104:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17104:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 2485 previous similar messages
Lustre: 16824:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 501 < left 1400, rollback = 2
Lustre: 16824:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 8 previous similar messages
Lustre: 16824:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/6, destroy: 1/4/0
Lustre: 16824:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1016 previous similar messages
Lustre: 16824:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 15/1400/0
Lustre: 16824:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1016 previous similar messages
Lustre: 16824:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 10/106/0
Lustre: 16824:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1016 previous similar messages
Lustre: 16824:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 2/5/1
Lustre: 16824:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1016 previous similar messages
Lustre: 16824:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 2/2/0
Lustre: 16824:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1016 previous similar messages
sched: RT throttling activated
Lustre: 14392:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 494 < left 582, rollback = 2
Lustre: 14392:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 15069:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1132, rollback = 7
Lustre: 15069:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1016 previous similar messages
LustreError: 11148:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000404:0x36:0x0]: rc = -2
LustreError: 12814:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration.
Lustre: 13882:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 511 < left 19684, rollback = 9
Lustre: 13882:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 18002:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18002:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 14214 previous similar messages
Lustre: 18002:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 18002:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 14214 previous similar messages
Lustre: 18002:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1583/1592/0, punch: 0/0/0, quota 3/3/0
Lustre: 18002:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 14214 previous similar messages
Lustre: 18002:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18002:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 14214 previous similar messages
Lustre: 18002:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18002:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 14213 previous similar messages
Lustre: 18002:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 511 < left 1591, rollback = 7
Lustre: 18002:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 14179 previous similar messages
Lustre: 15069:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 505 < left 624, rollback = 2
Lustre: 11102:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 510 < left 9791, rollback = 9
Lustre: 11102:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 10 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88027749e940/0xbfe3d35722953d53 lrc: 3/0,0 mode: PW/PW res: [0x340000401:0x170:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xbfe3d35722953d30 expref: 3530 pid: 18022 timeout: 410 lvb_type: 0
LustreError: 14919:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880256fa4500 x1710112928228224/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 19504:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802843aa5c0 x1710112928253952/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 19504:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 9 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 167-0: lustre-OST0002-osc-ffff88029ad2a548: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: 19536:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880254f97700 x1710112928305088/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
Lustre: 13087:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8802792c5180 x1710112926604672/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:469/0 lens 440/0 e 0 to 0 dl 1630891069 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18719:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880262e90680 x1710112926640384/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:470/0 lens 440/0 e 0 to 0 dl 1630891070 ref 2 fl New:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 18719:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 42 previous similar messages
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x56:0x0]// may get corrupted (rc -108)
Lustre: 19504:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880287e23ec0 x1710112926866304/t0(0) o101->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:473/0 lens 328/0 e 0 to 0 dl 1630891073 ref 2 fl New:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 19504:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 175 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88028aaf6d00/0xbfe3d3572293f923 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xbfe3d3572293f915 expref: 3266 pid: 12675 timeout: 419 lvb_type: 0
LustreError: 18324:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630891075 with bad export cookie 13827127652153247927
LustreError: 17755:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880297486fc8 ns: filter-lustre-OST0003_UUID lock: ffff8802675fcf00/0xbfe3d357229c40b7 lrc: 3/0,0 mode: --/PW res: [0x380000400:0x235:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x50000000020000 nid: 0@lo remote: 0xbfe3d357229c405c expref: 3075 pid: 17755 timeout: 0 lvb_type: 0
LustreError: 11-0: lustre-OST0003-osc-ffff880298db12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff880298db12a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff880298db12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 18719:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8802638abec0 x1710112926991488/t0(0) o101->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:477/0 lens 328/0 e 0 to 0 dl 1630891077 ref 2 fl New:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 18719:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 440 previous similar messages
LustreError: 20399:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029ad2a548: namespace resource [0x340000401:0xf:0x0].0x0 (ffff880081428f40) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880271f18400/0xbfe3d357229547f6 lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0xbfe3d357229547ef expref: 3775 pid: 17760 timeout: 420 lvb_type: 0
LustreError: 20399:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029ad2a548: namespace resource [0x340000401:0x1f:0x0].0x0 (ffff880271019bc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 12690:0:(ldlm_lockd.c:1342:ldlm_handle_enqueue0()) ### lock on disconnected export ffff880298c76fc8 ns: filter-lustre-OST0003_UUID lock: ffff88007d5dcf00/0xbfe3d357229c6c9a lrc: 2/0,0 mode: --/PR res: [0x380000400:0x5a7:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0xbfe3d357229c5e2a expref: -99 pid: 12690 timeout: 0 lvb_type: 0
LustreError: 19453:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/11s ago req@ffff880088e30cc0 x1710112926602880/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:469/0 lens 440/0 e 0 to 0 dl 1630891069 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 19453:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/11s); client may timeout req@ffff880088e30cc0 x1710112926602880/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:469/0 lens 440/0 e 0 to 0 dl 1630891069 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 11-0: lustre-OST0003-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff88029ad2a548: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
------------[ cut here ]------------
LustreError: 10977:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880258ab8040 x1710112928810304/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
WARNING: CPU: 6 PID: 11133 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
LustreError: 167-0: lustre-OST0003-osc-ffff88029ad2a548: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm crct10dif_pclmul ata_piix crct10dif_common drm_panel_orientation_quirks crc32c_intel serio_raw virtio_blk i2c_core libata floppy
CPU: 6 PID: 11133 Comm: mdt_rdpg03_000 Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0c004ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0bf19eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0bf4417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0c14aa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0c14ad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0ca9d08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04c459d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04c8cf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0e61af4>] mdt_mfd_close+0x1f4/0x870 [mdt]
[<ffffffffa0e67e14>] mdt_close_internal+0xb4/0x240 [mdt]
[<ffffffffa0e68224>] mdt_close+0x284/0x850 [mdt]
[<ffffffffa07cb2d4>] ? tgt_request_preprocess.isra.27+0xf4/0x850 [ptlrpc]
[<ffffffffa07cc26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032345e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0777990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffff810a20ad>] ? mod_timer+0x1dd/0x260
[<ffffffffa0779529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0778950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 58aae1581d9a010e ]---
LustreError: 20550:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff880298db12a8: namespace resource [0x380000400:0x10:0x0].0x0 (ffff880280d791c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20550:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 891 previous similar messages
Lustre: ll_ost06_011: service thread pid 18717 was inactive for 62.178 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 18717, comm: ll_ost06_011 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
Lustre: 17263:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 505 < left 849, rollback = 2
Lustre: 17263:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2 previous similar messages
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 18715, comm: ll_ost06_010 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 18042, comm: ll_ost06_006 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost06_001: service thread pid 12685 was inactive for 62.924 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: ll_ost06_020: service thread pid 19430 was inactive for 62.144 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 9 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff880266e861c0/0xbfe3d357229661fc lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x389:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xbfe3d357229661bd expref: 3128 pid: 17901 timeout: 433 lvb_type: 0
LustreError: 14920:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88025a62b240 x1710112929303872/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 10965:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630891090 with bad export cookie 13827127652153247913
LustreError: 11-0: lustre-OST0001-osc-ffff880298db12a8: operation ost_getattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff880298db12a8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 4122:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x96:0x0]// may get corrupted (rc -108)
LustreError: 167-0: lustre-OST0001-osc-ffff880298db12a8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: ll_ost06_053: service thread pid 19545 was inactive for 62.024 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 5 previous similar messages
Lustre: 4122:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000404:0x2c:0x0]/ may get corrupted (rc -108)
Lustre: 19536:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (69/2s); client may timeout req@ffff8802635fb880 x1710112926612608/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:495/0 lens 440/432 e 1 to 0 dl 1630891095 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 19536:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 42 previous similar messages
LustreError: 19504:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 68/6s ago req@ffff8802c626d428 x1710112926620864/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:494/0 lens 440/0 e 1 to 0 dl 1630891094 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 19504:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 42 previous similar messages
Lustre: 19504:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (68/6s); client may timeout req@ffff8802c626d428 x1710112926620864/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:494/0 lens 440/0 e 1 to 0 dl 1630891094 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 19504:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 7 previous similar messages
LustreError: 20691:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88029ad2a548: namespace resource [0x380000401:0xf:0x0].0x0 (ffff8802716247c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20691:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 667 previous similar messages
Lustre: 4126:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630891036/real 1630891036] req@ffff880258888cc0 x1710112927241728/t0(0) o101->lustre-OST0000-osc-ffff880298db12a8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630891091 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: lustre-OST0000-osc-ffff880298db12a8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0000: Client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:1876 to 0x340000400:1921
Lustre: 19506:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (43/25s); client may timeout req@ffff8802572ff700 x1710112926874176/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:479/0 lens 440/0 e 0 to 0 dl 1630891079 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 19703:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 43/26s ago req@ffff8802572f8680 x1710112926874304/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:479/0 lens 440/0 e 0 to 0 dl 1630891079 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 19703:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 76 previous similar messages
Lustre: 11129:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 509 < left 24735, rollback = 7
Lustre: 11129:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 11793 previous similar messages
Lustre: 11129:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 575/2300/0, destroy: 1/4/1
Lustre: 11129:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 11827 previous similar messages
Lustre: 11129:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 575/0/0
Lustre: 11129:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 11827 previous similar messages
Lustre: 11129:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 2876/24735/0, punch: 0/0/0, quota 10/74/0
Lustre: 11129:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 11827 previous similar messages
Lustre: 11129:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 575/9775/0, delete: 2/5/1
Lustre: 11129:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 11827 previous similar messages
Lustre: 11129:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/1
Lustre: 11129:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 11827 previous similar messages
LustreError: 20811:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880298db12a8: namespace resource [0x300000400:0x10:0x0].0x0 (ffff880280d78540) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20811:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 994 previous similar messages
LustreError: 19469:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 68/14s ago req@ffff880264319300 x1710112926621632/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:494/0 lens 440/0 e 1 to 0 dl 1630891094 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
LustreError: 19469:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 429 previous similar messages
Lustre: 19504:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (71/10s); client may timeout req@ffff88026b0d7700 x1710112926644992/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:499/0 lens 440/0 e 1 to 0 dl 1630891099 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 19504:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 720 previous similar messages
Lustre: 19536:0:(service.c:1436:ptlrpc_at_send_early_reply()) @@@ Could not add any time (72/30), not sending early reply req@ffff880254907700 x1710112927606656/t0(0) o101->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:513/0 lens 328/0 e 1 to 0 dl 1630891182 ref 2 fl New:/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: lustre-OST0002: haven't heard from client 03033789-bf5a-4e49-a70e-41fbce44720e (at 0@lo) in 53 seconds. I think it's dead, and I am evicting it. exp ffff8800823b1bf8, cur 1630891122 expire 1630891092 last 1630891069
Lustre: lustre-OST0003: haven't heard from client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff8802f3898008, cur 1630891123 expire 1630891093 last 1630891076
LustreError: 20399:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029ad2a548: namespace resource [0x340000400:0x30:0x0].0x0 (ffff88026c7d11c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20399:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 753 previous similar messages
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: 17204:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 511 < left 25872, rollback = 9
Lustre: 17204:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 19483:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (93/6s); client may timeout req@ffff880297b870c0 x1710112926586368/t0(0) o1->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:518/0 lens 440/432 e 2 to 0 dl 1630891118 ref 1 fl Complete:/0/0 rc 0/0 job:'lfs.0'
Lustre: 19483:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 1810 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 4 previous similar messages
LustreError: Skipped 3 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff88029ad2a548: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
INFO: task lfs:17451 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
lfs D ffff88029595f3c0 11360 17451 16290 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff8139a526>] ima_file_check+0xa6/0x1b0
[<ffffffff81255397>] do_last+0x587/0x1280
[<ffffffff812546de>] ? link_path_walk+0x27e/0x8c0
[<ffffffff812570ad>] path_openat+0xcd/0x5b0
[<ffffffff81258dfd>] do_filp_open+0x4d/0xb0
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e326e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81267893>] ? __alloc_fd+0xc3/0x170
[<ffffffff812447e4>] do_sys_open+0x124/0x220
[<ffffffff812448fe>] SyS_open+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task chown:18381 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
chown D ffff88027597bd58 12760 18381 16588 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff81257b2d>] ? putname+0x3d/0x60
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81242ddd>] chown_common.isra.15+0xed/0x1b0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff8124436e>] SyS_fchownat+0xce/0x120
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task touch:19035 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
touch D ffff88028d282980 12944 19035 17362 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81254bd0>] link_path_walk+0x770/0x8c0
[<ffffffff81254924>] link_path_walk+0x4c4/0x8c0
[<ffffffff8124818c>] ? get_empty_filp+0x5c/0x1f0
[<ffffffff8125708e>] path_openat+0xae/0x5b0
[<ffffffff81258dfd>] do_filp_open+0x4d/0xb0
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e326e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81267893>] ? __alloc_fd+0xc3/0x170
[<ffffffff812447e4>] do_sys_open+0x124/0x220
[<ffffffff812448fe>] SyS_open+0x1e/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task setfattr:19037 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
setfattr D ffff880290098040 12904 19037 16225 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff81270cc3>] SyS_removexattr+0x43/0xd0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task ln:19056 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ln D ffff880283e6d2c0 12760 19056 16462 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff8124b6b3>] vfs_fstatat+0x63/0xc0
[<ffffffff8124ba7e>] SYSC_newstat+0x2e/0x60
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8124bf4e>] SyS_newstat+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task truncate:19074 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
truncate D ffff8802744fb760 12904 19074 17245 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff817d5bb2>] lookup_slow+0x33/0xa7
[<ffffffff81256d6e>] path_lookupat+0x89e/0x8d0
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257b9f>] ? getname_flags+0x4f/0x1a0
[<ffffffff81256dcb>] filename_lookup+0x2b/0xc0
[<ffffffff81258cc7>] user_path_at_empty+0x67/0xc0
[<ffffffff817e8655>] ? __do_page_fault+0x1c5/0x500
[<ffffffff81258d31>] user_path_at+0x11/0x20
[<ffffffff81243914>] do_sys_truncate+0x44/0xb0
[<ffffffff81243b0e>] SyS_truncate+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mkdir:19083 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mkdir D ffff880292f1ca80 11824 19083 16395 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81256ee5>] filename_create+0x85/0x180
[<ffffffff8121f126>] ? kmem_cache_alloc+0x126/0x2e0
[<ffffffff81257c14>] ? getname_flags+0xc4/0x1a0
[<ffffffff81257ec1>] user_path_create+0x41/0x60
[<ffffffff81259206>] SyS_mkdirat+0x46/0xe0
[<ffffffff812592b9>] SyS_mkdir+0x19/0x20
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task file_exec.sh:19090 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
file_exec.sh D ffff88029009c240 12832 19090 17373 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff812550a6>] do_last+0x296/0x1280
[<ffffffff812546de>] ? link_path_walk+0x27e/0x8c0
[<ffffffff812570ad>] path_openat+0xcd/0x5b0
[<ffffffff810c18ca>] ? prepare_creds+0x3a/0x200
[<ffffffff8121daba>] ? ___slab_alloc+0x17a/0x560
[<ffffffff81258dfd>] do_filp_open+0x4d/0xb0
[<ffffffff810c18ca>] ? prepare_creds+0x3a/0x200
[<ffffffff810c1592>] ? creds_are_invalid.part.1+0x12/0x50
[<ffffffff810c1592>] ? creds_are_invalid.part.1+0x12/0x50
[<ffffffff810c1a20>] ? prepare_creds+0x190/0x200
[<ffffffff8124db04>] do_open_exec+0x24/0x110
[<ffffffff8124e171>] do_execve_common.isra.22+0x221/0x850
[<ffffffff81257c00>] ? getname_flags+0xb0/0x1a0
[<ffffffff8124ea49>] SyS_execve+0x29/0x30
[<ffffffff817ee5a8>] stub_execve+0x48/0x80
INFO: task rm:19108 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Lustre: lustre-OST0001: haven't heard from client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp ffff88024d5637e8, cur 1630891138 expire 1630891108 last 1630891090
Lustre: Skipped 1 previous similar message
rm D ffff88029516c240 12760 19108 16009 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff8125828b>] do_unlinkat+0x13b/0x2b0
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff812592fb>] SyS_unlinkat+0x1b/0x40
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 11119:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '3' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 3' to finish migration.
Lustre: 17842:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630891104/real 1630891104] req@ffff880078760cc0 x1710112928249408/t0(0) o101->lustre-OST0000-osc-ffff880298db12a8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630891148 ref 2 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'truncate.0'
Lustre: 17842:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 153 previous similar messages
Lustre: lustre-OST0000-osc-ffff880298db12a8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0000: Client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) reconnecting
Lustre: Skipped 4 previous similar messages
Lustre: lustre-OST0000-osc-ffff880298db12a8: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 4126:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630891104/real 1630891104] req@ffff8802976dbec0 x1710112929364096/t0(0) o101->lustre-OST0002-osc-ffff880298db12a8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630891148 ref 1 fl Rpc:XQr/2/ffffffff rc -11/-1 job:'getfattr.0'
Lustre: lustre-OST0002: Client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:884 to 0x300000401:929
LustreError: 20691:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88029ad2a548: namespace resource [0x380000400:0x30:0x0].0x0 (ffff88026c7d2ac0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 20691:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 2006 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff880298db12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4114:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880262fb2c00 x1710112934239296/t0(0) o10->lustre-OST0000-osc-ffff88029ad2a548@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891240 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 11141:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 12s req@ffff880257dd25c0 x1710112933969280/t0(0) o37->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:0/0 lens 448/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'ls.0'
LustreError: 16706:0:(mdt_xattr.c:429:mdt_dir_layout_update()) lustre-MDT0001: [0x240000403:0xcd:0x0] migrate mdt index mismatch 1 != 0
Lustre: 16952:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 511 < left 5711, rollback = 9
Lustre: 16952:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 14232:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '11' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 11' to finish migration.
LustreError: 22406:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880298db12a8: inode [0x280000403:0x1d:0x0] mdc close failed: rc = -13
LustreError: 19174:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0002: '19' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush 19' to finish migration.
Lustre: dir [0x240000403:0x90:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x240000403:0x90:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: 4118:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630891103/real 1630891103] req@ffff880263da70c0 x1710112930103552/t0(0) o101->lustre-OST0000-osc-ffff880298db12a8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630891162 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 4118:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 219 previous similar messages
Lustre: 4114:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880262fb2c00 x1710112935982784/t0(0) o10->lustre-OST0000-osc-ffff88029ad2a548@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891263 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'cp.0'
Lustre: 21323:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/9, destroy: 0/0/0
Lustre: 21323:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 37397 previous similar messages
Lustre: 21323:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 11/877/0
Lustre: 21323:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 37397 previous similar messages
Lustre: 21323:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 21323:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 37397 previous similar messages
Lustre: 21323:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 21323:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 37397 previous similar messages
Lustre: 21323:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 21323:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 37397 previous similar messages
Lustre: 11108:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 41548, rollback = 7
Lustre: 11108:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 37387 previous similar messages
Lustre: 4118:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630891103/real 1630891103] req@ffff880285938cc0 x1710112930099840/t0(0) o101->lustre-OST0000-osc-ffff880298db12a8@0@lo:28/4 lens 328/400 e 0 to 1 dl 1630891245 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'ln.0'
Lustre: 4118:0:(client.c:2285:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
INFO: task rm:19009 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff88025fb624f0 11648 19009 16036 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff8125828b>] do_unlinkat+0x13b/0x2b0
[<ffffffff817df62e>] ? mutex_unlock+0xe/0x10
[<ffffffff81293556>] ? dnotify_flush+0x46/0x110
[<ffffffff812592fb>] SyS_unlinkat+0x1b/0x40
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 11145:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0x40:0x0]: rc = -2
LustreError: 16667:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration.
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff88027abfb4c0/0xbfe3d35722a44156 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0xa1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000030020 nid: 0@lo remote: 0xbfe3d35722a44148 expref: 1664 pid: 17778 timeout: 634 lvb_type: 0
LustreError: 12663:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88025cadc500 x1710112938763968/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 10965:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630891292 with bad export cookie 13827127652154235151
LustreError: 11-0: lustre-OST0003-osc-ffff880298db12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 12 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298db12a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 7 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff880298db12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298db12a8: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0003-osc-ffff880298db12a8: disconnect after 20s idle
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff880255499a80/0xbfe3d35722a58ccc lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x8fe:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xbfe3d35722a58c71 expref: 5206 pid: 19474 timeout: 669 lvb_type: 0
LustreError: 19549:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88025a764b40 x1710112939829184/t0(0) o106->lustre-OST0003@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 19549:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 6 previous similar messages
LustreError: 167-0: lustre-OST0003-osc-ffff88029ad2a548: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4113:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x17b:0x0]// may get corrupted (rc -108)
Lustre: 4114:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x190:0x0]// may get corrupted (rc -108)
Lustre: 19286:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 503 < left 699, rollback = 2
Lustre: 19286:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 81 previous similar messages
LustreError: 24815:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88029ad2a548: namespace resource [0x380000400:0xc9b:0x0].0x0 (ffff880275f7d1c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 24815:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 3018 previous similar messages
Lustre: lustre-OST0003-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: 4116:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880254e9a5c0 x1710112940903040/t0(0) o10->lustre-OST0003-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891407 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
LustreError: 16692:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration.
Lustre: 26422:0:(service.c:1378:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880297b91300 x1710112943369024/t4294984811(0) o10->0bf25ca2-fb17-4410-82fe-9fe450342b30@0@lo:76/0 lens 440/432 e 0 to 0 dl 1630891431 ref 1 fl Complete:/0/0 rc 0/0 job:'dir_create.sh.0'
Lustre: 26422:0:(service.c:1378:ptlrpc_at_send_early_reply()) Skipped 176 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8800822cbc40/0xbfe3d35722adbe79 lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x630:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xbfe3d35722adbe4f expref: 1442 pid: 24490 timeout: 804 lvb_type: 0
LustreError: 14919:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88025a684500 x1710112945278464/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 14919:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 58 previous similar messages
LustreError: 11-0: lustre-OST0003-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff88029ad2a548: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: 167-0: lustre-OST0003-osc-ffff88029ad2a548: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4121:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88007873ea80 x1710112945368960/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891497 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
LustreError: 27076:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0003-osc-ffff88029ad2a548: namespace resource [0x380000400:0x15fa:0x0].0x0 (ffff880260d9d440) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 27076:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 105 previous similar messages
Lustre: lustre-OST0003-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
ptlrpc_watchdog_fire: 28 callbacks suppressed
Lustre: ll_ost05_004: service thread pid 17755 was inactive for 62.055 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 11 previous similar messages
Lustre: ll_ost05_038: service thread pid 24474 was inactive for 62.044 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 24474, comm: ll_ost05_038 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 24603, comm: ll_ost05_055 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 18398, comm: ll_ost05_011 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: 14232:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/7, destroy: 0/0/0
Lustre: 14232:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 131683 previous similar messages
Lustre: 14232:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 15/1383/0
Lustre: 14232:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 131683 previous similar messages
Lustre: 14232:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 14232:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 131683 previous similar messages
Lustre: 14232:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/2, delete: 0/0/0
Lustre: 14232:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 131683 previous similar messages
Lustre: 14232:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 14232:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 131683 previous similar messages
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 510 < left 80721, rollback = 7
Lustre: 16897:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 146576 previous similar messages
Lustre: 4120:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88029a09d180 x1710112947480768/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891546 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4120:0:(client.c:1485:after_reply()) Skipped 131 previous similar messages
LustreError: 12293:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -116
LustreError: 12293:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -116
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff88025d7a5680/0xbfe3d35722afdf06 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x161c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->393215) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xbfe3d35722afdef1 expref: 3068 pid: 19703 timeout: 863 lvb_type: 0
LustreError: 10967:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630891519 with bad export cookie 13827127652154240093
LustreError: 10967:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
Lustre: 4124:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x153:0x0]// may get corrupted (rc -108)
Lustre: 4124:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000404:0xe8:0x0]// may get corrupted (rc -108)
Lustre: 4121:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88007873ea80 x1710112947776000/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891588 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4121:0:(client.c:1485:after_reply()) Skipped 45 previous similar messages
LustreError: 25320:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802553a8680 x1710112947977920/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 25320:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 20 previous similar messages
LustreError: 1301:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli 0bf25ca2-fb17-4410-82fe-9fe450342b30/ffff8802f3898958 has 1269760 pending on destroyed export
LustreError: 27388:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 203407424 != fo_tot_granted 204677184
LustreError: 27388:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 1301:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 203404736 != fo_tot_granted 204674496
LustreError: 1301:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12697:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 211816640 != fo_tot_granted 213086400
LustreError: 12697:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 896 != fo_tot_pending 1270656
LustreError: 12697:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 211814848 != fo_tot_granted 213084608
LustreError: 12697:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 896 != fo_tot_pending 1270656
LustreError: 12708:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 235987392 != fo_tot_granted 237257152
LustreError: 12708:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12708:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 5120000 != fo_tot_pending 6389760
LustreError: 12708:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 230864704 != fo_tot_granted 232134464
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12710:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12710:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
Lustre: 4121:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000404:0x66:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001: haven't heard from client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff88024ade8008, cur 1630891566 expire 1630891536 last 1630891519
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 230865728 != fo_tot_granted 232135488
LustreError: 11057:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_obd_disconnect: tot_pending 0 != fo_tot_pending 1269760
LustreError: 11057:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 6 previous similar messages
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 7 previous similar messages
Lustre: 4124:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88026c3f25c0 x1710112950350528/t0(0) o10->lustre-OST0003-osc-ffff88029ad2a548@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891595 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
Lustre: 4124:0:(client.c:1485:after_reply()) Skipped 71 previous similar messages
Lustre: lustre-OST0003: haven't heard from client 0bf25ca2-fb17-4410-82fe-9fe450342b30 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880277fe9bf8, cur 1630891590 expire 1630891560 last 1630891543
Lustre: 17390:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880258662c00 x1710112950832384/t0(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:0/0 lens 304/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'mdt03_002.0'
LustreError: 12707:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 239756736 != fo_tot_granted 241026496
LustreError: 12707:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 18 previous similar messages
LustreError: 12707:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12707:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 19 previous similar messages
Lustre: dir [0x200000403:0xbf:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 18488:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '15' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 15' to finish migration.
Lustre: 4121:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88007873ea80 x1710112951886912/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630891691 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4121:0:(client.c:1485:after_reply()) Skipped 15 previous similar messages
Lustre: 16694:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 498 < left 2646, rollback = 2
Lustre: 16694:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 101 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 120s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880257fbda40/0xbfe3d35722af87f6 lrc: 3/0,0 mode: PW/PW res: [0x340000401:0x64e:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->1572863) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0xbfe3d35722af87ef expref: 3857 pid: 18297 timeout: 987 lvb_type: 0
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) Skipped 1 previous similar message
LustreError: 10957:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630891643 with bad export cookie 13827127652154084938
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: 167-0: lustre-OST0002-osc-ffff88029ad2a548: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: Skipped 2 previous similar messages
Lustre: 4118:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x2d1:0x0]/ may get corrupted (rc -108)
Lustre: 4119:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x304:0x0]// may get corrupted (rc -108)
Lustre: 4119:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000404:0xe2:0x0]// may get corrupted (rc -108)
Lustre: 16897:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8800ab324ad8 x1710112953235328/t0(0) o35->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'ls.0'
Lustre: 12709:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff88025a5ff700 x1710112953387328/t0(0) o13->lustre-MDT0002-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp-pre-2-2.0'
LustreError: 19692:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88026e0c2c00 x1710112953751104/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 19692:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 186 previous similar messages
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
LustreError: 12695:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 239759808 != fo_tot_granted 241029568
LustreError: 12695:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 33 previous similar messages
LustreError: 12695:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12695:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 33 previous similar messages
Lustre: 4121:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880078739f80 x1710112954727104/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 1 to 0 dl 1630891707 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4121:0:(client.c:1485:after_reply()) Skipped 102 previous similar messages
LustreError: 15473:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1630891419, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0000_UUID lock: ffff880265f1ed00/0xbfe3d35722afe6d1 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x179e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 15473 timeout: 0 lvb_type: 0
LustreError: 24477:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1630891419, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0000_UUID lock: ffff880252ab5a40/0xbfe3d35722afe97f lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x17a3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 24477 timeout: 0 lvb_type: 0
LustreError: 24477:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) Skipped 3 previous similar messages
LustreError: dumping log to /tmp/lustre-log.1630891720.15473
LustreError: 24526:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1630891420, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0000_UUID lock: ffff88027b18f480/0xbfe3d35722afe9a9 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x17a9:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 24526 timeout: 0 lvb_type: 0
LustreError: 24526:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) Skipped 16 previous similar messages
LustreError: 19604:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1630891419, 304s ago); not entering recovery in server code, just going back to sleep ns: filter-lustre-OST0000_UUID lock: ffff88026c82a200/0xbfe3d35722afe6c3 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x179d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x40010000000000 nid: local remote: 0x0 expref: -99 pid: 19604 timeout: 0 lvb_type: 0
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff88007a11d680/0xbfe3d35722be3be9 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x1a4d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xbfe3d35722be3bdb expref: 2217 pid: 18042 timeout: 1131 lvb_type: 0
LustreError: 11138:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0000: failed to get lu_attr of [0x200000403:0xbf:0x0]: rc = -2
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 245590592 != fo_tot_granted 246860352
LustreError: 12710:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 36 previous similar messages
LustreError: 12710:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12710:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 36 previous similar messages
Lustre: 4116:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000403:0x7e:0x0]/ may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x2a0:0x0]// may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x389:0x0]// may get corrupted (rc -108)
Lustre: 4116:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000403:0x177:0x0]/ may get corrupted (rc -108)
LustreError: 31975:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029ad2a548: namespace resource [0x340000400:0x23b0:0x0].0x0 (ffff880272268cc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31975:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 419 previous similar messages
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: 4119:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802774a7700 x1710112961831168/t0(0) o10->lustre-OST0000-osc-ffff88029ad2a548@0@lo:6/4 lens 440/432 e 1 to 0 dl 1630891907 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 4119:0:(client.c:1485:after_reply()) Skipped 69 previous similar messages
LustreError: 16897:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0002: failed to get lu_attr of [0x280000403:0x119:0x0]: rc = -2
Lustre: dir [0x240000403:0x2e6:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
ptlrpc_watchdog_fire: 43 callbacks suppressed
Lustre: mdt05_000: service thread pid 11110 was inactive for 64.137 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
Pid: 11110, comm: mdt05_000 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_reint_object_lock+0x2c/0x60 [mdt]
[<0>] mdt_reint_striped_lock+0x89/0x5b0 [mdt]
[<0>] mdt_reint_unlink+0x8dd/0x19b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: dir [0x240000403:0x25f:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: mdt05_007: service thread pid 19174 was inactive for 64.245 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 19174, comm: mdt05_007 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_reint_object_lock+0x2c/0x60 [mdt]
[<0>] mdt_reint_striped_lock+0x89/0x5b0 [mdt]
[<0>] mdt_reint_unlink+0x8dd/0x19b0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 11433:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 11433:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 11433:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11433:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: mdt02_005: service thread pid 16952 was inactive for 90.121 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: mdt02_004: service thread pid 16630 was inactive for 90.128 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 45 previous similar messages
Pid: 16952, comm: mdt02_005 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_object_local_lock+0x527/0xb90 [mdt]
[<0>] mdt_object_lock_internal+0x70/0x390 [mdt]
[<0>] mdt_getattr_name_lock+0xaca/0x2590 [mdt]
[<0>] mdt_intent_getattr+0x2d5/0x4b0 [mdt]
[<0>] mdt_intent_opc+0x1dd/0xc10 [mdt]
[<0>] mdt_intent_policy+0x1a1/0x360 [mdt]
[<0>] ldlm_lock_enqueue+0x3c2/0xb40 [ptlrpc]
[<0>] ldlm_handle_enqueue0+0xaa6/0x16d0 [ptlrpc]
[<0>] tgt_enqueue+0x64/0x240 [ptlrpc]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 11910:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11910:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message
LustreError: 11910:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 11910:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 1 previous similar message
Lustre: 20785:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x449:0x0] with magic=0xbd60bd0
Lustre: 11101:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1083/4332/0, destroy: 0/0/0
Lustre: 11101:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 243566 previous similar messages
Lustre: 11101:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 1085/15/0
Lustre: 11101:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 243564 previous similar messages
Lustre: 11101:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5416/46579/0, punch: 0/0/0, quota 14/174/10
Lustre: 11101:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 243565 previous similar messages
Lustre: 11101:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 1083/18411/0, delete: 0/0/0
Lustre: 11101:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 243565 previous similar messages
Lustre: 11101:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11101:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 243567 previous similar messages
Lustre: 25851:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 505 < left 1202, rollback = 7
Lustre: 25851:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 239958 previous similar messages
Lustre: 26936:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x44b:0x0] with magic=0xbd60bd0
Lustre: 26936:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
LustreError: 12701:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 283486016 != fo_tot_granted 284755776
LustreError: 12701:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 75 previous similar messages
LustreError: 12701:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 1269760
LustreError: 12701:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 75 previous similar messages
LustreError: 10978:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: ffff880253c152c0/0xbfe3d35722c9734e lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x2337:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xbfe3d35722c97347 expref: 1524 pid: 19693 timeout: 1411 lvb_type: 0
LustreError: 18063:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802612e5e00 x1710112970065408/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 18063:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 55 previous similar messages
LustreError: 11-0: lustre-OST0002-osc-ffff88029ad2a548: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 3 previous similar messages
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
LustreError: 2736:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88029ad2a548: inode [0x280000403:0x1d:0x0] mdc close failed: rc = -13
LustreError: 167-0: lustre-OST0001-osc-ffff88029ad2a548: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 26991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630892068 with bad export cookie 13827127652153255963
LustreError: 26991:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 2 previous similar messages
LustreError: Skipped 3 previous similar messages
Lustre: 4123:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x20a:0x0]/ may get corrupted (rc -108)
Lustre: 4122:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x467:0x0]// may get corrupted (rc -108)
Lustre: 4122:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x362:0x0]/ may get corrupted (rc -108)
Lustre: 4123:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x351:0x0]/ may get corrupted (rc -108)
LustreError: 24678:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0002: cli 03033789-bf5a-4e49-a70e-41fbce44720e/ffff88007ca9c138 has 811008 pending on destroyed export
LustreError: 17019:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630892091 with bad export cookie 13827127652156651355
LustreError: 2801:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0002-osc-ffff88029ad2a548: namespace resource [0x340000400:0x2340:0x0].0x0 (ffff88027720dbc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 2801:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 995 previous similar messages
Lustre: 4121:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x456:0x0]/ may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x3e8:0x0]// may get corrupted (rc -108)
Lustre: 4120:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x324:0x0]/ may get corrupted (rc -108)
Lustre: 4121:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x367:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: 4119:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff88027f13ac00 x1710112971279040/t0(0) o10->lustre-OST0000-osc-ffff880298db12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630892179 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
Lustre: 4119:0:(client.c:1485:after_reply()) Skipped 2 previous similar messages
Lustre: 22996:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x475:0x0] with magic=0xbd60bd0
Lustre: 22996:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 16952:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x445:0x0] with magic=0xbd60bd0
Lustre: 16952:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 20785:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0002-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x280000403:0x4d1:0x0] with magic=0xbd60bd0
Lustre: 20785:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 499 < left 1339, rollback = 2
Lustre: 11111:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 193 previous similar messages
Lustre: 20785:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x356:0x0] with magic=0xbd60bd0
Lustre: 20785:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 3 previous similar messages
Lustre: 21323:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x35e:0x0] with magic=0xbd60bd0
Lustre: 21323:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 1 previous similar message
Lustre: 19322:0:(lod_lov.c:1225:lod_parse_striping()) lustre-MDT0001-mdtlov: EXTENSION flags=40 set on component[2]=1 of non-SEL file [0x240000403:0x3ab:0x0] with magic=0xbd60bd0
Lustre: 19322:0:(lod_lov.c:1225:lod_parse_striping()) Skipped 9 previous similar messages
Lustre: 4114:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x3e8:0x0]/ may get corrupted (rc -108)
Lustre: 4113:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000403:0x255:0x0]// may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000403:0x279:0x0]/ may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x54c:0x0]/ may get corrupted (rc -108)
Lustre: 4116:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000404:0x314:0x0]// may get corrupted (rc -108)
Lustre: 4113:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x42e:0x0]// may get corrupted (rc -108)
ptlrpc_watchdog_fire: 15 callbacks suppressed
Lustre: ll_ost00_046: service thread pid 25547 was inactive for 40.018 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 25547, comm: ll_ost00_046 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 23304, comm: ll_ost00_017 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Lustre: ll_ost00_007: service thread pid 17814 was inactive for 40.114 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 14 previous similar messages
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 25521, comm: ll_ost00_030 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] tgt_extent_lock+0xea/0x2a0 [ptlrpc]
[<0>] ofd_getattr_hdl+0x365/0x720 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
LustreError: 21296:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '14' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 14' to finish migration.
LustreError: 5431:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000404:0x3a9:0x0] error: rc = -5
LustreError: 3670:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88029ad2a548: inode [0x280000403:0x4df:0x0] mdc close failed: rc = -5
LustreError: 5431:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 12 previous similar messages
LustreError: 5455:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 2683:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0002-mdc-ffff88029ad2a548: [0x280000400:0x9:0x0] lock enqueue fails: rc = -108
Lustre: dir [0x280000404:0x13d:0x0] stripe 0 readdir failed: -108, directory is partially accessed!
Lustre: Skipped 1 previous similar message
Lustre: ll_ost00_040: service thread pid 25541 was inactive for 46.068 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 41 previous similar messages
Lustre: lustre-OST0001: haven't heard from client 03033789-bf5a-4e49-a70e-41fbce44720e (at 0@lo) in 52 seconds. I think it's dead, and I am evicting it. exp ffff880257a35d28, cur 1630892347 expire 1630892317 last 1630892295
Lustre: 4129:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x417:0x0]// may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x4fe:0x0]// may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000404:0x26e:0x0]// may get corrupted (rc -108)
Lustre: 4117:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x3da:0x0]/ may get corrupted (rc -108)
LustreError: 4139:0:(vvp_io.c:1793:vvp_io_init()) lustre: refresh file layout [0x280000403:0x4fe:0x0] error -108.
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000403:0x24c:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x3f1:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000403:0x55b:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x240000403:0x317:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x2ed:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000404:0x115:0x0]// may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x200000404:0x11d:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x38d:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff88029ad2a548: Connection restored to 192.168.123.130@tcp (at 0@lo)
Lustre: Skipped 5 previous similar messages
LustreError: 30634:0:(service.c:2290:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-0@lo: deadline 25/98s ago req@ffff88029bb070c0 x1710112976388544/t0(0) o1->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:196/0 lens 440/0 e 0 to 0 dl 1630892306 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 25522:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (25/98s); client may timeout req@ffff8802588557c0 x1710112976388864/t0(0) o1->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:196/0 lens 440/0 e 0 to 0 dl 1630892306 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'lfs.0'
Lustre: 25522:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 6 previous similar messages
LustreError: 30634:0:(service.c:2290:ptlrpc_server_handle_request()) Skipped 1888 previous similar messages
LustreError: 1301:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 03033789-bf5a-4e49-a70e-41fbce44720e/ffff880298c39bf8 has 122880 pending on destroyed export
Lustre: 4119:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x3aa:0x0]// may get corrupted (rc -108)
Lustre: 4119:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.130@tcp:/lustre/fid: [0x280000404:0x3a8:0x0]/ may get corrupted (rc -108)
LustreError: 11097:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88029966ca88 ns: mdt-lustre-MDT0002_UUID lock: ffff880266ee61c0/0xbfe3d35722d822e8 lrc: 3/0,0 mode: PR/PR res: [0x280000404:0x3a9:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xbfe3d35722d822cc expref: 16 pid: 11097 timeout: 0 lvb_type: 0
Lustre: 16648:0:(service.c:2328:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (88/28s); client may timeout req@ffff8802751dc500 x1710112976853248/t0(0) o101->03033789-bf5a-4e49-a70e-41fbce44720e@0@lo:269/0 lens 576/640 e 0 to 0 dl 1630892379 ref 1 fl Complete:/0/0 rc -107/-107 job:'ls.0'
Lustre: 16648:0:(service.c:2328:ptlrpc_server_handle_request()) Skipped 13 previous similar messages
LustreError: 11119:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88029966ca88 ns: mdt-lustre-MDT0002_UUID lock: ffff880286197480/0xbfe3d35722d7ffe1 lrc: 3/0,0 mode: PR/PR res: [0x280000404:0x3a9:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xbfe3d35722d7ff94 expref: 5 pid: 11119 timeout: 0 lvb_type: 0
LustreError: 11119:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) Skipped 4 previous similar messages
Lustre: lustre-OST0003-osc-ffff88029ad2a548: disconnect after 36s idle
LustreError: 11095:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88029966ca88 ns: mdt-lustre-MDT0002_UUID lock: ffff88026221cf00/0xbfe3d35722d828fa lrc: 3/0,0 mode: PR/PR res: [0x280000404:0x3a9:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xbfe3d35722d82883 expref: 2 pid: 11095 timeout: 0 lvb_type: 0
------------[ cut here ]------------
WARNING: CPU: 11 PID: 11111 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm crct10dif_pclmul ata_piix crct10dif_common drm_panel_orientation_quirks crc32c_intel serio_raw virtio_blk i2c_core libata floppy
CPU: 11 PID: 11111 Comm: mdt05_001 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0c2f262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0c0f33d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0c0f437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0cd3e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04ca6f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa04896b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa0476760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa047c4ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04735ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04a4a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa080d60d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa07dd892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0f0ef3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0de3408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0dc4b2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04c9375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0dc5b86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0e970d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0e56e53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0e57087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0e2c55c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0e39657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07cc26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032345e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0777990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa0779529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0778950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 58aae1581d9a010f ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-2): ldiskfs_getblk:888: inode #231: block 51071: comm mdt05_001: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-2-8.
LDISKFS-fs (dm-2): Remounting filesystem read-only
LustreError: 11111:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0002: error reading offset 278528 (block 68, size 24, offs 278496), credits 0/0: rc = -28
LustreError: 11111:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0002-osd: write updates failed: rc = -28
LustreError: 11111:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 11111:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0002: can't update reply_data file: rc = -30
LustreError: 11111:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0002: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-2) in osd_trans_stop:2082: error 28
LustreError: 13847:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0002-osc-MDT0002: fail to cancel 24 llog-records: rc = -30
LustreError: 14484:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0003-osc-MDT0002: can't cancel 26 records: rc = -30
LustreError: 13844:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0001-osc-MDT0002: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 13844:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0001-osc-MDT0002: can't cancel record: rc = -30
LustreError: 11111:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0002: failed to stop transaction: rc = -28
LustreError: 11111:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0002-osd: stop trans failed: rc = -30
LustreError: 12248:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff88024feb4948 commit error: 2
LustreError: 8536:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LDISKFS-fs error (device dm-2) in osd_trans_stop:2082: IO failure
LustreError: 13847:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 7 previous similar messages
LustreError: 13847:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0002-osc-MDT0002: can't cancel record: rc = -30
LustreError: 13847:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 7 previous similar messages
LustreError: 13062:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0000-osc-MDT0002: can't cancel 5 records: rc = -30
LustreError: 13062:0:(osp_sync.c:1094:osp_sync_process_committed()) Skipped 12 previous similar messages

BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa1016e52>] ll_migrate+0x9b2/0xec0 [lustre]
PGD 80000002867d9067 PUD 292cdf067 PMD 0
Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy
CPU: 1 PID: 32579 Comm: lfs Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880321528010 ti: ffff880260dac000 task.ti: ffff880260dac000
RIP: 0010:[<ffffffffa1016e52>] [<ffffffffa1016e52>] ll_migrate+0x9b2/0xec0 [lustre]
RSP: 0018:ffff880260dafbc8 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff8800764623f8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8803272ca038 RDI: ffff8803272ca018
RBP: ffff880260dafc48 R08: ffff880249700bb8 R09: ffff880260dafa60
R10: 0000000000000006 R11: 0000000000000007 R12: 0000000000000000
R13: ffff88027c335cc8 R14: ffff880268131cd8 R15: 0000000000000030
FS: 00007fabd0075740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 00000002852da000 CR4: 00000000001607e0
Call Trace:
[<ffffffff81242213>] ? __check_object_size+0x1c3/0x220
[<ffffffffa1004561>] ll_dir_ioctl+0x5d01/0x6ed0 [lustre]
[<ffffffff8103de0f>] ? save_stack_trace+0x1f/0x30
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817d5822>] ? free_debug_processing+0x240/0x272
[<ffffffffa10326ad>] ? ll_finish_md_op_data+0xad/0x1f0 [lustre]
[<ffffffff8121e474>] ? __slab_free+0xa4/0x300
[<ffffffff811becd5>] ? __free_pages+0x25/0x30
[<ffffffffa0ffaf3e>] ? ll_release_page+0x8e/0xa0 [lustre]
[<ffffffffa04b3a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e326e>] ? _raw_spin_unlock+0xe/0x20
[<ffffffff81277ec8>] ? __mark_inode_dirty+0x1a8/0x2c0
[<ffffffff8125b3fd>] do_vfs_ioctl+0x40d/0x6c0
[<ffffffff817df62e>] ? mutex_unlock+0xe/0x10
[<ffffffff8125b81a>] ? iterate_dir+0xaa/0x120
[<ffffffff8125b751>] SyS_ioctl+0xa1/0xc0
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24

Lustre: lustre-OST0000-osc-ffff88008dee12a8: disconnect after 21s idle
LustreError: 11516:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 llog-records: rc = -2
LustreError: 11516:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 17175:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 511 < left 8447, rollback = 9
Lustre: 17175:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 495/1980/0, destroy: 1/4/0
Lustre: 17175:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 503/283/0
Lustre: 17175:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 2476/21295/0, punch: 0/0/0, quota 6/6/0
Lustre: 17175:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 497/8447/0, delete: 3/6/1
Lustre: 17175:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 17175:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 8: before 510 < left 8447, rollback = 9
Lustre: 17175:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 495/1980/0, destroy: 1/4/0
Lustre: 17175:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 5/5/1, xattr_set: 503/283/0
Lustre: 17175:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 2476/21295/0, punch: 0/0/0, quota 6/6/0
Lustre: 17175:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 497/8447/0, delete: 3/6/1
Lustre: 17175:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 17175:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 508 < left 21295, rollback = 7
Lustre: 17406:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 508 < left 610, rollback = 2
Lustre: 17406:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/2, destroy: 0/0/0
Lustre: 17406:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17406:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 8/610/0
Lustre: 17406:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17406:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0
Lustre: 17406:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17406:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/2, delete: 1/1/0
Lustre: 17406:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17406:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 17406:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 13880:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 504 < left 671, rollback = 2
Lustre: 13880:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 13880:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/4, destroy: 0/0/0
Lustre: 13880:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 13880:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/671/0
Lustre: 13880:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 13880:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 8/56/0
Lustre: 13880:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 13880:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 10/203/4, delete: 0/0/0
Lustre: 13880:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 13880:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0
Lustre: 13880:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 17478:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 511 < left 3020, rollback = 7
Lustre: 16667:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 500 < left 610, rollback = 2
Lustre: 16667:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 16667:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 3/12/8, destroy: 0/0/0
Lustre: 16667:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 287 previous similar messages
Lustre: 16667:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 8/610/0
Lustre: 16667:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 287 previous similar messages
Lustre: 16667:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0
Lustre: 16667:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 287 previous similar messages
Lustre: 16667:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/215/4, delete: 1/1/0
Lustre: 16667:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 287 previous similar messages
Lustre: 16667:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 1/1/0
Lustre: 16667:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 287 previous similar messages
Lustre: 11186:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 499 < left 1606, rollback = 2
Lustre: 11186:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 11186:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 2/8/8, destroy: 1/4/0
Lustre: 11186:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11186:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 19/1606/0
Lustre: 11186:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11186:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1/10/0, punch: 0/0/0, quota 8/104/0
Lustre: 11186:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11186:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 12/194/4, delete: 3/6/1
Lustre: 11186:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11186:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 7/7/0, ref_del: 3/3/0
Lustre: 11186:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 11175:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 504 < left 19317, rollback = 7
Lustre: 11175:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 279 previous similar messages
Lustre: 19140:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 510 < left 21295, rollback = 7
Lustre: 19140:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1800 previous similar messages
LustreError: 17580:0:(mdd_object.c:3461:mdd_close()) lustre-MDD0001: failed to get lu_attr of [0x240000404:0xc:0x0]: rc = -2
------------[ cut here ]------------
WARNING: CPU: 7 PID: 19140 at /home/green/git/lustre-release/ldiskfs/namei.c:3331 ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy
CPU: 7 PID: 19140 Comm: mdt_rdpg03_003 Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0bcd4ae>] ldiskfs_orphan_add+0x11e/0x2a0 [ldiskfs]
[<ffffffffa0bbe9eb>] ldiskfs_xattr_inode_orphan_add.constprop.17+0xbb/0x110 [ldiskfs]
[<ffffffffa0bc1417>] ldiskfs_xattr_delete_inode+0x77/0x3c0 [ldiskfs]
[<ffffffffa0be1aa0>] ? ldiskfs_evict_inode+0x190/0x670 [ldiskfs]
[<ffffffffa0be1ad9>] ldiskfs_evict_inode+0x1c9/0x670 [ldiskfs]
[<ffffffff812649af>] evict+0xaf/0x180
[<ffffffff81264de5>] iput+0xf5/0x180
[<ffffffffa0c76d08>] osd_object_delete+0x1f8/0x370 [osd_ldiskfs]
[<ffffffffa04d359d>] lu_object_free.isra.26+0xcd/0x1d0 [obdclass]
[<ffffffffa04d7cf1>] lu_object_put+0xa1/0x480 [obdclass]
[<ffffffffa0e0faf4>] mdt_mfd_close+0x1f4/0x870 [mdt]
[<ffffffffa0e15e14>] mdt_close_internal+0xb4/0x240 [mdt]
[<ffffffffa0e16224>] mdt_close+0x284/0x850 [mdt]
[<ffffffffa07da2d4>] ? tgt_request_preprocess.isra.27+0xf4/0x850 [ptlrpc]
[<ffffffffa07db26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032945e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0786990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa0788529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff817e32be>] ? _raw_spin_unlock_irq+0xe/0x30
[<ffffffffa0787950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 3e032eb585932b8d ]---
Lustre: 11168:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 509 < left 19317, rollback = 7
Lustre: 11168:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 1982 previous similar messages
Lustre: dir [0x200000404:0x21:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 20569:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20569:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 3828 previous similar messages
Lustre: 20569:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/89/0
Lustre: 20569:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 3828 previous similar messages
Lustre: 20569:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 626/635/0, punch: 0/0/0, quota 3/3/0
Lustre: 20569:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 3828 previous similar messages
Lustre: 20569:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20569:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 3828 previous similar messages
Lustre: 20569:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20569:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 3828 previous similar messages
LustreError: 11164:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '4' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 4' to finish migration.
Lustre: 16667:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 509 < left 1125, rollback = 7
Lustre: 16667:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 2277 previous similar messages
LustreError: 11349:0:(out_handler.c:910:out_tx_end()) lustre-MDT0002-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
LustreError: 11949:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11949:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 18661:0:(mdt_xattr.c:429:mdt_dir_layout_update()) lustre-MDT0001: [0x240000404:0x1e:0x0] migrate mdt index mismatch 1 != 0
Lustre: 16688:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 504 < left 26928, rollback = 7
Lustre: 16688:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 6320 previous similar messages
LustreError: 22103:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff88008dee12a8: inode [0x240000403:0xc1:0x0] mdc close failed: rc = -13
Lustre: 16718:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 509 < left 10658, rollback = 9
Lustre: 16718:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 43 previous similar messages
Lustre: 16718:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 626/2504/0, destroy: 1/4/0
Lustre: 16718:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 11107 previous similar messages
Lustre: 16718:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 630/148/0
Lustre: 16718:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 11107 previous similar messages
Lustre: 16718:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 3131/26928/0, punch: 0/0/0, quota 10/74/0
Lustre: 16718:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 11107 previous similar messages
Lustre: 16718:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 627/10658/0, delete: 2/5/1
Lustre: 16718:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 11107 previous similar messages
Lustre: 16718:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 16718:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 11107 previous similar messages
Lustre: ll_ost07_006: service thread pid 20126 was inactive for 40.063 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 20302, comm: ll_ost07_009 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: Skipped 1 previous similar message
Pid: 20126, comm: ll_ost07_006 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 20290, comm: ll_ost07_008 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] ofd_destroy_by_fid+0x1d1/0x520 [ofd]
[<0>] ofd_destroy_hdl+0x27f/0xa20 [ofd]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: ll_ost07_002: service thread pid 12722 was inactive for 40.506 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: 16806:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 507 < left 47955, rollback = 7
Lustre: 16806:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 3626 previous similar messages
Lustre: 21569:0:(client.c:2285:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1630890481/real 1630890481] req@ffff8802848f57c0 x1710112383577664/t0(0) o101->lustre-MDT0002-mdc-ffff880299a4ca88@0@lo:12/10 lens 576/33552 e 0 to 1 dl 1630890527 ref 2 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'ln.0'
Lustre: lustre-MDT0002-mdc-ffff880299a4ca88: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-MDT0002: Client 989be46e-abfb-4210-a299-8eaa2ef57189 (at 0@lo) reconnecting
Lustre: lustre-MDT0002-mdc-ffff880299a4ca88: Connection restored to 192.168.123.170@tcp (at 0@lo)
Lustre: ll_ost03_001: service thread pid 12705 was inactive for 62.121 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: ll_ost03_000: service thread pid 12704 was inactive for 62.012 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: 16812:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 501 < left 894, rollback = 2
Lustre: 16812:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 46 previous similar messages
Lustre: ll_ost03_016: service thread pid 21607 was inactive for 62.068 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 11 previous similar messages
Lustre: dir [0x240000404:0x56:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 6 previous similar messages
sched: RT throttling activated
Lustre: dir [0x240000403:0x3b:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x200000404:0x105:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 25120:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 473/1892/0, destroy: 1/4/0
Lustre: 25120:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 10812 previous similar messages
Lustre: 25120:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/1, xattr_set: 477/148/0
Lustre: 25120:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 10812 previous similar messages
Lustre: 25120:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 2366/20349/0, punch: 0/0/0, quota 10/90/0
Lustre: 25120:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 10812 previous similar messages
Lustre: 25120:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 474/8057/0, delete: 2/5/1
Lustre: 25120:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 10812 previous similar messages
Lustre: 17206:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17206:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 10812 previous similar messages
Lustre: mdt00_010: service thread pid 19511 was inactive for 62.001 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 9 previous similar messages
Lustre: 16679:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 7: before 510 < left 1678, rollback = 7
Lustre: 16679:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 11426 previous similar messages
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff880085b661c0/0x99ae7c71d2c4aef4 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x25:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->524287) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x99ae7c71d2c4acd2 expref: 2005 pid: 18408 timeout: 437 lvb_type: 0
LustreError: 11-0: lustre-OST0001-osc-ffff88008dee12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff88008dee12a8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 11-0: lustre-OST0001-osc-ffff88008dee12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 12712:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88008397f0c0 x1710112388083264/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 167-0: lustre-OST0001-osc-ffff88008dee12a8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 17633:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802705a5180 x1710112388122944/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 26071:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff88008dee12a8: namespace resource [0x100:0x0:0x0].0x0 (ffff880274faa0c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 17633:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Lustre: 11170:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 509 < left 6391, rollback = 9
Lustre: 11170:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 43 previous similar messages
Lustre: dir [0x200000403:0xb9:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: 4139:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802594b4500 x1710112388789568/t0(0) o10->lustre-OST0003-osc-ffff880299a4ca88@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630890657 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff88008344f480/0x99ae7c71d2c89f17 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x99ae7c71d2c89f09 expref: 124 pid: 11175 timeout: 459 lvb_type: 0
LustreError: 11-0: lustre-MDT0001-mdc-ffff880299a4ca88: operation mds_reint to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff880299a4ca88: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-MDT0001-mdc-ffff880299a4ca88: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 19511:0:(ldlm_lockd.c:1425:ldlm_handle_enqueue0()) ### lock on destroyed export ffff880298c95d28 ns: mdt-lustre-MDT0001_UUID lock: ffff880083f4cb40/0x99ae7c71d2c8a14e lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x1e:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x99ae7c71d2c8a124 expref: 9 pid: 19511 timeout: 0 lvb_type: 0
LustreError: 22976:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 22526:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x1e:0x0] error: rc = -5
LustreError: 22976:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 22525:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0001-mdc-ffff880299a4ca88: [0x240000400:0x6:0x0] lock enqueue fails: rc = -108
LustreError: 25916:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x1e:0x0] error: rc = -108
LustreError: 25916:0:(file.c:5042:ll_inode_revalidate_fini()) Skipped 35 previous similar messages
LustreError: 23067:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 23067:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 1 previous similar message
LustreError: 22517:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880299a4ca88: inode [0x240000404:0x1e:0x0] mdc close failed: rc = -108
LustreError: 24645:0:(file.c:5042:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x1e:0x0] error: rc = -108
LustreError: 23925:0:(mdc_request.c:1436:mdc_read_page()) lustre-MDT0001-mdc-ffff880299a4ca88: [0x240000400:0xa:0x0] lock enqueue fails: rc = -108
LustreError: 23925:0:(mdc_request.c:1436:mdc_read_page()) Skipped 6 previous similar messages
LustreError: 26512:0:(file.c:234:ll_close_inode_openhandle()) lustre-clilmv-ffff880299a4ca88: inode [0x240000404:0x85:0x0] mdc close failed: rc = -108
LustreError: 26512:0:(file.c:234:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Lustre: lustre-MDT0001-mdc-ffff880299a4ca88: Connection restored to 192.168.123.170@tcp (at 0@lo)
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff880264eaa980/0x99ae7c71d2ca562d lrc: 3/0,0 mode: PW/PW res: [0x259:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 2097152->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x99ae7c71d2ca55d9 expref: 2594 pid: 21804 timeout: 477 lvb_type: 0
LustreError: 21784:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880271884b40 x1710112389583168/t0(0) o106->lustre-OST0000@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0000-osc-ffff880299a4ca88: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff880299a4ca88: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 4122:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff880281b55e00 x1710112389660928/t0(0) o10->lustre-OST0003-osc-ffff88008dee12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630890646 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dir_create.sh.0'
INFO: task mv:20961 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff880087920040 11040 20961 16069 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:22715 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff8802935fc9d0 11168 22715 16174 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
INFO: task mv:23043 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff88027c4ac9d0 11344 23043 16557 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
Lustre: lustre-OST0001: haven't heard from client 099f8133-efe2-4519-9be0-b3a78691d601 (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp ffff8800ab1bdd28, cur 1630890633 expire 1630890603 last 1630890586
LustreError: 167-0: lustre-OST0000-osc-ffff880299a4ca88: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: lustre-OST0000: Client 989be46e-abfb-4210-a299-8eaa2ef57189 (at 0@lo) reconnecting
LustreError: 676:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0000: cli 989be46e-abfb-4210-a299-8eaa2ef57189/ffff88008b3f5d28 has 28672 pending on destroyed export
Lustre: lustre-OST0001-osc-ffff88008dee12a8: Connection restored to 192.168.123.170@tcp (at 0@lo)
LustreError: 676:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 108219584 != fo_tot_granted 108248256
LustreError: 676:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 0 != fo_tot_pending 28672
LustreError: 12726:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 108219584 != fo_tot_granted 108248256
LustreError: 12726:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
Lustre: 4135:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x280000404:0x116:0x0]// may get corrupted (rc -108)
LustreError: 12736:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 108218688 != fo_tot_granted 108247360
LustreError: 12736:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
Lustre: lustre-OST0001-osc-ffff88008dee12a8: disconnect after 56s idle
Lustre: Skipped 7 previous similar messages
LustreError: 11-0: lustre-OST0001-osc-ffff88008dee12a8: operation ost_disconnect to node 0@lo failed: rc = -107
LustreError: Skipped 55 previous similar messages
LustreError: 27236:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880299a4ca88: namespace resource [0x258:0x0:0x0].0x0 (ffff880265aea0c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 27236:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 91 previous similar messages
Lustre: lustre-OST0000-osc-ffff880299a4ca88: Connection restored to 192.168.123.170@tcp (at 0@lo)
LustreError: 12736:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 108218304 != fo_tot_granted 108246976
LustreError: 12724:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 896 != fo_tot_pending 29568
LustreError: 12724:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 12736:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 2 previous similar messages
LustreError: 12735:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 116655424 != fo_tot_granted 116684096
LustreError: 12735:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 1 previous similar message
LustreError: 12735:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 61440 != fo_tot_pending 90112
LustreError: 12735:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 2 previous similar messages
Lustre: 4128:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802596e3240 x1710112390445824/t0(0) o10->lustre-OST0003-osc-ffff88008dee12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630890712 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
LustreError: 12735:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 116595008 != fo_tot_granted 116623680
LustreError: 12735:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 5 previous similar messages
LustreError: 12735:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 12735:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 5 previous similar messages
LustreError: 12736:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 125033408 != fo_tot_granted 125062080
LustreError: 12736:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 5 previous similar messages
LustreError: 12736:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 28672
LustreError: 12736:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 5 previous similar messages
Lustre: 11180:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0
Lustre: 11180:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 54542 previous similar messages
Lustre: 11180:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 11180:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 54542 previous similar messages
Lustre: 11180:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1377/1386/0, punch: 0/0/0, quota 8/80/0
Lustre: 11180:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 54542 previous similar messages
Lustre: 11180:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0
Lustre: 11180:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 54433 previous similar messages
Lustre: 11180:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11180:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 54542 previous similar messages
Lustre: 4128:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802596e3240 x1710112392380992/t0(0) o10->lustre-OST0003-osc-ffff88008dee12a8@0@lo:6/4 lens 440/432 e 0 to 0 dl 1630890763 ref 1 fl Rpc:RQU/2/0 rc 0/-115 job:'dd.0'
Lustre: 25409:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 511 < left 6761, rollback = 7
Lustre: 25409:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 55169 previous similar messages
LustreError: 12730:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 125032768 != fo_tot_granted 125061440
LustreError: 12730:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 13 previous similar messages
LustreError: 12730:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 896 != fo_tot_pending 29568
LustreError: 12730:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 13 previous similar messages
Lustre: 16808:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0001: opcode 8: before 509 < left 23408, rollback = 9
Lustre: 16808:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 78 previous similar messages
LustreError: 19511:0:(mdd_object.c:401:mdd_xattr_get()) lustre-MDD0001: object [0x240000403:0x1b8:0x0] not found: rc = -2
LustreError: 23151:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 200271168 != fo_tot_granted 200299840
LustreError: 23151:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 33 previous similar messages
LustreError: 23151:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 896 != fo_tot_pending 29568
LustreError: 23151:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 33 previous similar messages
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff88007afee940/0x99ae7c71d2cfaf86 lrc: 3/0,0 mode: PW/PW res: [0x7fa:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x99ae7c71d2cfaf7f expref: 2008 pid: 19847 timeout: 641 lvb_type: 0
LustreError: 14707:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880298f957c0 x1710112395697792/t0(0) o106->lustre-OST0001@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 11-0: lustre-OST0001-osc-ffff88008dee12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff88008dee12a8: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 4117:0:(osc_request.c:1035:osc_init_grant()) lustre-OST0001-osc-ffff88008dee12a8: granted 4419584 but already consumed 6287360
LustreError: 167-0: lustre-OST0001-osc-ffff88008dee12a8: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 14707:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 13 previous similar messages
LustreError: 11344:0:(out_handler.c:910:out_tx_end()) lustre-MDT0002-osd: undo for /home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/out_handler.c:659: rc = -524
LustreError: 11344:0:(out_handler.c:910:out_tx_end()) Skipped 1 previous similar message
LustreError: 11949:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 llog-records: rc = -116
LustreError: 11949:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0002-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116
Lustre: 4137:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x200000403:0x14a:0x0]/ may get corrupted (rc -108)
Lustre: 4137:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x240000403:0x1fe:0x0]// may get corrupted (rc -108)
Lustre: 4136:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x200000403:0x11c:0x0]/ may get corrupted (rc -108)
Lustre: 4136:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x200000403:0x15d:0x0]/ may get corrupted (rc -108)
Lustre: 4136:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x280000403:0x20a:0x0]/ may get corrupted (rc -108)
LustreError: 30045:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff88008dee12a8: namespace resource [0x300000401:0xc6f:0x0].0x0 (ffff88007bda34c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 30045:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffff88008dee12a8: Connection restored to 192.168.123.170@tcp (at 0@lo)
Lustre: 11338:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff88028d2d25c0 x1710112396732672/t0(0) o1000->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 1528/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp_up1-0.0'
Lustre: dir [0x280000404:0x127:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 10 previous similar messages
LustreError: 11155:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '5' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush 5' to finish migration.
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff8802615b07c0/0x99ae7c71d2d620e6 lrc: 3/0,0 mode: PW/PW res: [0x82f:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x99ae7c71d2d620ca expref: 3007 pid: 12703 timeout: 724 lvb_type: 0
LustreError: 18556:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802864e9300 x1710112398720576/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 18556:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 10 previous similar messages
LustreError: 21853:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630890872 with bad export cookie 11073925362060566975
LustreError: 11008:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630890873 with bad export cookie 11073925362060566975
LustreError: 11008:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) Skipped 1 previous similar message
LustreError: 11-0: lustre-OST0000-osc-ffff880299a4ca88: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff880299a4ca88: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0000-osc-ffff880299a4ca88: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: ffff88027be0c3c0/0x99ae7c71d2d1d3a8 lrc: 3/0,0 mode: PW/PW res: [0x7fa:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0x99ae7c71d2d1d3a1 expref: 3624 pid: 26915 timeout: 742 lvb_type: 0
LustreError: 10994:0:(ldlm_lockd.c:2496:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1630890890 with bad export cookie 11073925362059591728
Lustre: lustre-OST0001-osc-ffff880299a4ca88: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0001-osc-ffff880299a4ca88: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 27098:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880262a79f80 x1710112399324672/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:''
LustreError: 27098:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 14 previous similar messages
Lustre: 4130:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x200000403:0x15d:0x0]/ may get corrupted (rc -108)
Lustre: 4128:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x280000404:0x209:0x0]/ may get corrupted (rc -108)
Lustre: 17478:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff8802734b9300 x1710112399235136/t0(0) o35->099f8133-efe2-4519-9be0-b3a78691d601@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'cat.0'
Lustre: 21298:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 8s req@ffff88027b19f700 x1710112399317952/t0(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:0/0 lens 1528/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp_up0-1.0'
Lustre: 21298:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages
Lustre: 4125:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x240000405:0xbd:0x0]/ may get corrupted (rc -108)
LustreError: 5:0:(ofd_obd.c:501:ofd_destroy_export()) lustre-OST0001: cli 989be46e-abfb-4210-a299-8eaa2ef57189/ffff88028ea5c138 has 3194880 pending on destroyed export
LustreError: 31947:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0001-osc-ffff880299a4ca88: namespace resource [0x3d0:0x0:0x0].0x0 (ffff880261e256c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31947:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 195 previous similar messages
Lustre: lustre-OST0001-osc-ffff880299a4ca88: Connection restored to 192.168.123.170@tcp (at 0@lo)
LustreError: 12732:0:(tgt_grant.c:248:tgt_grant_sanity_check()) ofd_statfs: tot_granted 15350976 != fo_tot_granted 18545856
LustreError: 12732:0:(tgt_grant.c:248:tgt_grant_sanity_check()) Skipped 49 previous similar messages
LustreError: 12732:0:(tgt_grant.c:251:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 3194880
LustreError: 12732:0:(tgt_grant.c:251:tgt_grant_sanity_check()) Skipped 49 previous similar messages
LustreError: 31536:0:(ldlm_resource.c:1124:ldlm_resource_complain()) lustre-OST0000-osc-ffff880299a4ca88: namespace resource [0x520:0x0:0x0].0x0 (ffff88025cdce840) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 31536:0:(ldlm_resource.c:1124:ldlm_resource_complain()) Skipped 509 previous similar messages
Lustre: 12727:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff88029842de00 x1710112400440960/t0(0) o13->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp-pre-0-0.0'
Lustre: 12727:0:(service.c:2157:ptlrpc_server_handle_req_in()) Skipped 7 previous similar messages
Lustre: 18579:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880266d9a5c0 x1710112400811328/t0(0) o1000->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 1320/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'osp_up2-0.0'
LustreError: 16716:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0001: '4' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 4' to finish migration.
LustreError: 11949:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -2
LustreError: 11949:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -2
Lustre: 11165:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0
Lustre: 11165:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 106211 previous similar messages
Lustre: 11165:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/209/0
Lustre: 11165:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 106211 previous similar messages
Lustre: 11165:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 713/722/0, punch: 0/0/0, quota 8/56/0
Lustre: 11165:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 106211 previous similar messages
Lustre: 11165:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0
Lustre: 11165:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 106211 previous similar messages
Lustre: 11165:0:(osd_handler.c:1898:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11165:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 106211 previous similar messages
Lustre: 11187:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 7: before 512 < left 1069, rollback = 7
Lustre: 11187:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 104223 previous similar messages
INFO: task mv:30640 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mv D ffff8800823cc9d0 11216 30640 16676 0x00000080
Call Trace:
[<ffffffff817e19f9>] schedule_preempt_disabled+0x39/0x90
[<ffffffff817df80a>] __mutex_lock_slowpath+0x13a/0x340
[<ffffffff817dfa3d>] mutex_lock+0x2d/0x40
[<ffffffff81251fe1>] lock_rename+0x31/0xe0
[<ffffffff8125862f>] SYSC_renameat2+0x22f/0x570
[<ffffffff811ed432>] ? handle_mm_fault+0xc2/0x150
[<ffffffff817e8a76>] ? trace_do_page_fault+0x56/0x170
[<ffffffff8125975e>] SyS_renameat2+0xe/0x10
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
LustreError: 11026:0:(ldlm_lockd.c:259:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: ffff8800790b34c0/0x99ae7c71d2dd801c lrc: 3/0,0 mode: PW/PW res: [0x3a7:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000080020020 nid: 0@lo remote: 0x99ae7c71d2dd7ff9 expref: 3184 pid: 19978 timeout: 869 lvb_type: 0
LustreError: 11-0: lustre-OST0003-osc-ffff88008dee12a8: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 6 previous similar messages
Lustre: lustre-OST0003-osc-ffff88008dee12a8: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 167-0: lustre-OST0003-osc-ffff88008dee12a8: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 4122:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x240000403:0x2a6:0x0]/ may get corrupted (rc -108)
Lustre: 4123:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x280000403:0x2da:0x0]/ may get corrupted (rc -108)
Lustre: 4125:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x200000403:0x23b:0x0]/ may get corrupted (rc -108)
Lustre: 4121:0:(llite_lib.c:3298:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.170@tcp:/lustre/fid: [0x240000403:0x289:0x0]// may get corrupted (rc -5)
Lustre: 11198:0:(service.c:2157:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 6s req@ffff880260ba8680 x1710112404577984/t0(0) o35->099f8133-efe2-4519-9be0-b3a78691d601@0@lo:0/0 lens 392/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 job:'dir_create.sh.0'
Lustre: 11164:0:(osd_internal.h:1328:osd_trans_exec_op()) lustre-MDT0002: opcode 8: before 512 < left 23034, rollback = 9
Lustre: 11164:0:(osd_internal.h:1328:osd_trans_exec_op()) Skipped 237 previous similar messages
ptlrpc_watchdog_fire: 32 callbacks suppressed
Lustre: mdt01_008: service thread pid 17206 was inactive for 56.985 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
Pid: 17206, comm: mdt01_008 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_fini+0xa25/0xec0 [ptlrpc]
[<0>] ldlm_cli_enqueue+0x436/0xa30 [ptlrpc]
[<0>] osp_md_object_lock+0x160/0x300 [osp]
[<0>] lod_object_lock+0xdb/0x7d0 [lod]
[<0>] mdd_object_lock+0x2d/0xd0 [mdd]
[<0>] mdt_remote_object_lock_try+0x1cc/0x510 [mdt]
[<0>] mdt_remote_object_lock+0x2a/0x30 [mdt]
[<0>] mdt_rename_lock+0xbe/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: lustre-OST0003-osc-ffff88008dee12a8: Connection restored to 192.168.123.170@tcp (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: mdt01_004: service thread pid 16766 was inactive for 56.218 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 16766, comm: mdt01_004 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] ldlm_completion_ast+0x7a7/0xa00 [ptlrpc]
[<0>] ldlm_cli_enqueue_local+0x259/0x880 [ptlrpc]
[<0>] mdt_rename_lock+0x24b/0x4d0 [mdt]
[<0>] mdt_reint_migrate+0x6e8/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Pid: 11172, comm: mdt04_000 3.10.0-7.9-debug #2 SMP Wed Aug 4 02:24:01 EDT 2021
Call Trace:
[<0>] __cond_resched+0x26/0x30
[<0>] kmem_cache_alloc_trace+0x58/0x310
[<0>] sub_updates_write+0x759/0xe32 [ptlrpc]
[<0>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<0>] lod_trans_stop+0x25c/0x340 [lod]
[<0>] mdd_trans_stop+0x28/0x16e [mdd]
[<0>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<0>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<0>] mdo_migrate+0x4c/0x4e [mdt]
[<0>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<0>] mdt_reint_rec+0x87/0x240 [mdt]
[<0>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<0>] mdt_reint+0x67/0x150 [mdt]
[<0>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
Lustre: mdt04_011: service thread pid 25409 was inactive for 72.240 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
------------[ cut here ]------------
WARNING: CPU: 8 PID: 11172 at /home/green/git/lustre-release/ldiskfs/ext4_jbd2.c:266 __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
Modules linked in: loop zfs(PO) zunicode(PO) zzstd(O) zlua(O) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) jbd2 mbcache lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) dm_flakey dm_mod libcfs(OE) crc_t10dif crct10dif_generic sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy
CPU: 8 PID: 11172 Comm: mdt04_000 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
[<ffffffff817d9418>] dump_stack+0x19/0x1b
[<ffffffff8108d538>] __warn+0xd8/0x100
[<ffffffff8108d67d>] warn_slowpath_null+0x1d/0x20
[<ffffffffa0bfc262>] __ldiskfs_handle_dirty_metadata+0x1d2/0x230 [ldiskfs]
[<ffffffffa0bdc33d>] ldiskfs_getblk+0x13d/0x210 [ldiskfs]
[<ffffffffa0bdc437>] ldiskfs_bread+0x27/0xe0 [ldiskfs]
[<ffffffffa0ca0e97>] osd_write+0x4e7/0xcb0 [osd_ldiskfs]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa04d96f3>] dt_record_write+0x33/0x120 [obdclass]
[<ffffffffa04986b2>] llog_osd_write_rec+0x1052/0x19a0 [obdclass]
[<ffffffffa0485760>] llog_write_rec+0x290/0x590 [obdclass]
[<ffffffffa048b4ee>] llog_cat_add_rec+0x23e/0x950 [obdclass]
[<ffffffffa04825ff>] llog_add+0x17f/0x1f0 [obdclass]
[<ffffffffa04b3a39>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[<ffffffffa081c60d>] sub_updates_write+0xb6b/0xe32 [ptlrpc]
[<ffffffffa07ec892>] top_trans_stop+0x7c2/0xf60 [ptlrpc]
[<ffffffffa0ebcf3c>] lod_trans_stop+0x25c/0x340 [lod]
[<ffffffffa0d91408>] mdd_trans_stop+0x28/0x16e [mdd]
[<ffffffffa0d72b2f>] mdd_migrate_object+0xf1f/0x1ba0 [mdd]
[<ffffffffa04d8375>] ? lu_object_find_at+0x285/0xb10 [obdclass]
[<ffffffffa0d73b86>] mdd_migrate+0x3d6/0x8e0 [mdd]
[<ffffffffa0e450d8>] mdo_migrate+0x4c/0x4e [mdt]
[<ffffffffa0e04e53>] mdt_reint_migrate+0x1023/0x11d0 [mdt]
[<ffffffffa0e05087>] mdt_reint_rec+0x87/0x240 [mdt]
[<ffffffffa0dda55c>] mdt_reint_internal+0x76c/0xb40 [mdt]
[<ffffffffa0de7657>] mdt_reint+0x67/0x150 [mdt]
[<ffffffffa07db26e>] tgt_request_handle+0x83e/0x1870 [ptlrpc]
[<ffffffffa032945e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[<ffffffffa0786990>] ptlrpc_server_handle_request+0x250/0xb10 [ptlrpc]
[<ffffffffa0788529>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc]
[<ffffffff81411979>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffffa0787950>] ? ptlrpc_wait_event+0x620/0x620 [ptlrpc]
[<ffffffff810ba0f4>] kthread+0xe4/0xf0
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba010>] ? kthread_create_on_node+0x140/0x140
---[ end trace 3e032eb585932b8e ]---
LDISKFS-fs: ldiskfs_getblk:888: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
LDISKFS-fs error (device dm-0): ldiskfs_getblk:888: inode #193: block 52433: comm mdt04_000: journal_dirty_metadata failed: handle type 0 started at line 1972, credits 512/0, errcode -28
Aborting journal on device dm-0-8.
LDISKFS-fs (dm-0): Remounting filesystem read-only
LustreError: 11172:0:(osd_io.c:2103:osd_ldiskfs_write_record()) lustre-MDT0000: error reading offset 233472 (block 57, size 48, offs 233464), credits 0/0: rc = -28
LustreError: 11172:0:(update_trans.c:985:top_trans_stop()) lustre-MDT0000-osd: write updates failed: rc = -28
LustreError: 11172:0:(osd_io.c:2110:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30
LustreError: 11172:0:(tgt_lastrcvd.c:1253:tgt_add_reply_data()) lustre-MDT0000: can't update reply_data file: rc = -30
LustreError: 11172:0:(osd_handler.c:2075:osd_trans_stop()) lustre-MDT0000: failed in transaction hook: rc = -30
LDISKFS-fs error (device dm-0) in osd_trans_stop:2082: error 28
LustreError: 14938:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-OST0003-osc-MDT0000: fail to cancel 1 llog-records: rc = -30
LustreError: 14938:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-OST0003-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30
LustreError: 14938:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0003-osc-MDT0000: can't cancel record: rc = -30
LustreError: 11172:0:(osd_handler.c:2085:osd_trans_stop()) lustre-MDT0000: failed to stop transaction: rc = -28
LustreError: 11172:0:(update_trans.c:1011:top_trans_stop()) lustre-MDT0000-osd: stop trans failed: rc = -30
LustreError: 2510:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -5
LustreError: 10954:0:(osd_handler.c:1779:osd_trans_commit_cb()) transaction @0xffff8802c82744a8 commit error: 2
LustreError: 20203:0:(llog_cat.c:602:llog_cat_add_rec()) llog_write_rec -116: lh=ffff8802995589f8
LustreError: 20203:0:(update_trans.c:1062:top_trans_stop()) lustre-MDT0000-osp-MDT0001: write updates failed: rc = -116
LustreError: 11174:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '10' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 10' to finish migration.
LustreError: 2444:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 2444:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 4 previous similar messages
LustreError: 11174:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0002: [0x280000404:0x264:0x0] doesn't exist!: rc = -14
LustreError: 14938:0:(osp_sync.c:1079:osp_sync_process_committed()) lustre-OST0003-osc-MDT0000: can't cancel record: rc = -30
LustreError: 14938:0:(osp_sync.c:1079:osp_sync_process_committed()) Skipped 1 previous similar message
LustreError: 14241:0:(osp_sync.c:1094:osp_sync_process_committed()) lustre-OST0002-osc-MDT0000: can't cancel 4 records: rc = -30
LustreError: 15996:0:(llite_lib.c:1836:ll_md_setattr()) md_setattr fails: rc = -30
LustreError: 15996:0:(llite_lib.c:1836:ll_md_setattr()) Skipped 5 previous similar messages
LustreError: 16581:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0001: [0x240000405:0x178:0x0] doesn't exist!: rc = -14
LustreError: 12360:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 llog-records: rc = -2
LustreError: 12360:0:(llog_cat.c:751:llog_cat_cancel_arr_rec()) Skipped 11 previous similar messages
LustreError: 12360:0:(llog_cat.c:787:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -2
LustreError: 12360:0:(llog_cat.c:787:llog_cat_cancel_records()) Skipped 10 previous similar messages
LustreError: 21581:0:(mdt_open.c:1237:mdt_cross_open()) lustre-MDT0001: [0x240000405:0x178:0x0] doesn't exist!: rc = -14
LustreError: 17132:0:(mdd_dir.c:4121:mdd_migrate_cmd_check()) lustre-MDD0000: '4' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush 4' to finish migration.